2008-09-05T01:29:15 *** Mark__T has joined #openils-evergreen 2008-09-05T04:58:42 *** skmurphy_ has joined #openils-evergreen 2008-09-05T05:00:26 *** Mark__T has quit IRC 2008-09-05T05:00:26 *** rsinger__ has quit IRC 2008-09-05T05:00:26 *** jeff has quit IRC 2008-09-05T05:00:26 *** sarabee has quit IRC 2008-09-05T05:00:26 *** sylvar has quit IRC 2008-09-05T05:00:26 *** skmurphy has quit IRC 2008-09-05T05:00:26 *** kbeswick_ has quit IRC 2008-09-05T05:00:26 *** skmurphy_ is now known as skmurphy 2008-09-05T05:00:50 *** sarabee has joined #openils-evergreen 2008-09-05T05:02:24 *** kbeswick_ has joined #openils-evergreen 2008-09-05T05:06:33 *** jeff has joined #openils-evergreen 2008-09-05T05:07:21 *** sylvar has joined #openils-evergreen 2008-09-05T05:07:37 *** rsinger has joined #OpenILS-Evergreen 2008-09-05T05:11:58 *** kados has quit IRC 2008-09-05T05:12:11 *** kados has joined #openils-evergreen 2008-09-05T05:19:57 *** Mark__T has joined #openils-evergreen 2008-09-05T07:59:40 @later tell dbs no, there's no seed data needed in config.circ_matrix_test. If there is a matching matchpoint (there is a catch-all seed matchpoint ... are you installing from scratch or upgrading?) then the default is success. we may need to ask berick about the ML though (that's an ML event). (however, making me look did bring me to a different bug) 2008-09-05T07:59:40 miker_: The operation succeeded. 2008-09-05T08:12:16 *** rsinger_ has joined #OpenILS-Evergreen 2008-09-05T08:29:26 *** rsinger has quit IRC 2008-09-05T08:31:55 *** rsinger_ is now known as rsinger 2008-09-05T08:41:34 *** dbs has joined #openils-evergreen 2008-09-05T08:43:45 wow. whine on IRC late at night, get your whines answered early in the morning 2008-09-05T08:44:04 *** kgs has joined #openils-evergreen 2008-09-05T08:44:09 berick++ 2008-09-05T08:44:13 miker_++ 2008-09-05T08:44:21 (it's a daily ritual) 2008-09-05T08:45:07 bah 2008-09-05T08:45:48 doublebah 2008-09-05T08:47:51 kgs++ 2008-09-05T08:47:54 * kgs oh wait... 2008-09-05T08:50:33 miker_: I'll give your patch a shot - there is one row in ccmm 2008-09-05T09:06:55 *** Mark__T has left #openils-evergreen 2008-09-05T10:27:10 dbs 2008-09-05T10:27:19 waiting on the revised document, if you got the time =) 2008-09-05T10:34:27 * dbs looks for more time 2008-09-05T10:44:36 :) Thanks, dbs 2008-09-05T10:44:50 I'm hoping to get this wrapped up while I'm still working here 2008-09-05T10:46:02 skmurphy: how much longer are you around, short-timer? 2008-09-05T10:46:46 The 19th would be my last day, but I'm thinking about ducking out earlier for a camping trip 2008-09-05T10:46:56 Depends on when i get documents done ;) 2008-09-05T10:47:27 * dbs feels even guiltier 2008-09-05T10:48:04 hehe, don't worry about it, but i'd appreciate it when you get the chance... i think shae and kgs are going to send me some revisions, too 2008-09-05T10:49:12 skmurphy: dbs sent me the IRC log and as soon as I swim through my other stuff today I promise to give you feedback... 2008-09-05T10:49:19 would love to see templating :) 2008-09-05T10:49:25 i can send you waht i have so far 2008-09-05T10:49:27 or 2008-09-05T10:49:30 though I also think creating a slew of templates for EVERYONE is key 2008-09-05T10:49:31 if you can wait till the end of today 2008-09-05T10:49:39 skmurphy trust me I can wait ;-) 2008-09-05T10:49:42 i thought the PINES templates were to be shared with all EG deals? 2008-09-05T11:15:43 *** kgs is now known as kgs_writing 2008-09-05T11:18:31 Now that's odd... "DBD::Pg::st execute failed: ERROR: creation of Perl function failed: Can't locate JSON/XS.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at line 2", and yet find / -name XS.pm definitely turns up /usr/local/lib/perl/5.8.8/JSON/XS.pm... 2008-09-05T11:18:32 ...as one of the hits. And @INC includes /usr/local/lib/perl/5.8.8 -- so I wonder what's up... 2008-09-05T11:20:25 sylvar: did you restart pg after installing json::xs? if the perl interpreter (per backend) has already tried to load the module, it caches that state 2008-09-05T11:20:51 and yet, when I use an opensrf.xml file that points to "localhost" instead of "evergreen-db.library.utah.edu", the problem goes away. 2008-09-05T11:21:01 (localhost is evergreen-test, not evergreen.db) 2008-09-05T11:21:19 s/\./-/ 2008-09-05T11:21:20 different db server, you mean? 2008-09-05T11:21:23 yes 2008-09-05T11:21:54 the module is required on the application server, the db server and the apache server ... if they're all different 2008-09-05T11:21:56 and basically what I did was to change four instances of 'localhost' to the postgres server's fqdn, omitting references to memcached. 2008-09-05T11:22:24 ah, so I need to make sure JSON::XS is installed on the DB server. 2008-09-05T11:22:28 yes 2008-09-05T11:22:29 miker++ 2008-09-05T11:22:44 it's used inside a stored proc in 1.2.2 and beyond 2008-09-05T11:27:17 my, it's nice to have CPAN on a mirror on the same campus... hasn't been that way for me since my ufl.edu days 2008-09-05T11:31:02 sylvar: MiniCPAN <3 <3 <3 2008-09-05T11:31:23 anyone remember the name of the package that provides perl's headers and other development resources? JSON::XS can't compile. 2008-09-05T11:31:40 it doesn't seem to be perl-dev or dev-perl 2008-09-05T11:33:00 libperl-dev perhaps? 2008-09-05T11:33:33 miker++ again 2008-09-05T11:35:47 sylvar pasted "D'ohh!" at http://paste.lisp.org/display/66350 2008-09-05T11:36:38 is this a version mismatch? the JSON::XS compilation appears to be using 5.8, and the package appears to have installed it in a 5.8.8 directory 2008-09-05T11:37:37 sylvar: what distro is this? 2008-09-05T11:37:55 Ubuntu 8.04 2008-09-05T11:38:11 you should have an apt package for the module 2008-09-05T11:38:28 libjson-xs-perl or something 2008-09-05T11:42:23 libjson-xs-perl is only available for 'intrepid', which sounds like the sort of nonsense that installs postgres 8.3 by default. And I don't have physical access to the server anyway, so I'd rather not upgrade the OS from Atlanta if I can avoid it. Seems to be putting the cart before the horse at any rate. 2008-09-05T11:44:00 ah, and intrepid is a release that's not out yet. 2008-09-05T11:44:11 hrm... odd 2008-09-05T11:44:18 well, I'm stumped then :) 2008-09-05T11:44:37 (IOW, it's always built from CPAN for me...) 2008-09-05T11:50:20 that's a C compilation error 2008-09-05T11:50:26 XS modules are written in C 2008-09-05T11:50:42 looks like you don't have some development headers installed? 2008-09-05T11:50:59 wait, that was the libperl stuff 2008-09-05T11:51:00 just the conclusion I was reaching... I'm doing apt-get install build-essential now. 2008-09-05T11:51:01 ...catching up 2008-09-05T11:51:36 aaaaand... CPAN installed it! 2008-09-05T11:51:41 sboyette++ 2008-09-05T11:53:10 restarting postgres on db server 2008-09-05T11:57:33 sylvar pasted "A brand new error... that's what I call progress!" at http://paste.lisp.org/display/66352 2008-09-05T11:58:03 DBD::Pg::st execute failed: ERROR: could not access file "$libdir/tablefunc": No such file or directory 2008-09-05T12:00:57 any ideas on this one, O oracular sages? 2008-09-05T12:01:14 * sylvar really is doing a bit of veneration here 2008-09-05T12:02:30 postgresql contrib stuff 2008-09-05T12:02:44 you need the tablefunc contrib installed 2008-09-05T12:04:05 so that's a package that would have originated with postgres, right? 2008-09-05T12:05:29 not a package 2008-09-05T12:05:38 see the install instructions 2008-09-05T12:06:03 step 4 of http://open-ils.org/dokuwiki/doku.php?id=installing_evergreen_1.2_on_debian_etch_x86_32-bit 2008-09-05T12:06:45 ah, I guess that didn't get copied over when we dumped and restored the database form the old server. 2008-09-05T12:16:35 OK, got it installed (from 8.2, not 8.1 as the directions suggested), restarted postgres, and... 2008-09-05T12:16:39 DBD::Pg::st execute failed: no connection to the server [for Statement " SELECT ... 2008-09-05T12:17:34 do you have empirical evidence that the server is running? :) 2008-09-05T12:17:43 because occam's razor says it isn't 2008-09-05T12:17:44 yes, psql -h evergreen-db.library.utah.edu -U postgres evergreen works fine. 2008-09-05T12:18:29 ok. i'll defer to dbs or miker then :) 2008-09-05T12:18:33 I can connect from the OPAC/application server using that command, do "select * from permission.grp_tree;", and get a sensible result. 2008-09-05T12:19:12 Help me, dbs or miker_, you're my only hope. Both of you. 2008-09-05T12:19:56 -= THIS MESSAGE NOT LOGGED =- 2008-09-05T12:20:05 sylvar: restart_perl? restart_c? 2008-09-05T12:20:13 in fact, just restart_perl 2008-09-05T12:20:25 (DBD::Pg is the Perl database driver) 2008-09-05T12:20:34 sorry, I was on the phone for the past hour 2008-09-05T12:21:12 and yes, it's good to adjust the instructions to correspond to your distribution as you're using a not-really-officially supported distro :) 2008-09-05T12:22:20 fair enough :) 2008-09-05T12:22:51 OK, I tried again after restart_perl and got no error messages per se but also no results. gateway.log says osrf_json_gw 2008-09-05 10:21:55 [INFO:19515:osrf_app_session.c:142:1220630083195158] Returning NULL from app_request_recv after timeout 2008-09-05T12:24:13 router.log says: 2008-09-05T12:24:16 router 2008-09-05 10:23:41 [INFO:19437:transport_session.c:410:] Received message with type cancel and code 503 2008-09-05T12:24:17 router 2008-09-05 10:23:41 [INFO:19437:osrf_router.c:268:12203663371376614] Received network layer error message from evergreen@evergreen-test.library.utah.edu/open-ils.search_listener_at_evergreen-test.library.utah.edu_19706 2008-09-05T12:24:19 router 2008-09-05 10:23:41 [WARN:19437:osrf_router.c:275:12203663371376614] We lost the last node in the class, responding with error and removing... 2008-09-05T12:25:21 does your list of running OpenSRF services look good? 2008-09-05T12:26:24 ps aux | grep opensrf | wc -l gives the result "70 2008-09-05T12:26:41 er, "70". So I'm not sure what I should be looking for. 2008-09-05T12:26:58 On the app server, that is, not the DB server. 2008-09-05T12:27:46 no opensrf running on the db server. hope that's not a huge duh; I was under the impression that mostly the db server just needed postgres. 2008-09-05T12:27:58 no duh, that's correct 2008-09-05T12:28:02 whew 2008-09-05T12:28:08 you've tried some other search other than hydrogeology? 2008-09-05T12:28:23 industrial design, harry potter, steampunk 2008-09-05T12:28:44 just making sure it's not a cached result 2008-09-05T12:29:05 i dunno - restart apache? 2008-09-05T12:29:07 I've also tried rerunning the searches uncached by doing shift+[Reload] 2008-09-05T12:29:26 no no, that will pull from the results cached by memcached 2008-09-05T12:30:28 "steampunk -fruit" "steampunk -meat" "steampunk -veggies" will be three separate searches that avoid memcached 2008-09-05T12:31:50 ok, I restarted apache and tried "industrial music" and same result in router.log 2008-09-05T12:33:20 what about osrfsys.log and open-ils.search_unix.log? 2008-09-05T12:34:58 not seeing any activity in those logfiles when I submit new searches 2008-09-05T12:35:25 how's srfsh working for you? 2008-09-05T12:36:27 can you fire up srfsh and log in? or cut'n'paste a search request into it? 2008-09-05T12:36:28 Login failed. Dang it. I'm running it as opensrf, sending 'login admin [password]', and it's not working. 2008-09-05T12:37:03 This seems like a useful failure to notice, at least. :) 2008-09-05T12:37:05 Hmm. I'm a drama queen, but I would be tempted to just stop apache, stop_all, and start_router, start_perl, start_c, start apache 2008-09-05T12:37:16 just to get back to a clean slate 2008-09-05T12:37:41 and leave memcached intact? maybe I'm just a bigger drama queen than you are. or maybe I'm just a process princess. 2008-09-05T12:38:12 memcached shouldn't need to be restarted 2008-09-05T12:38:30 ok 2008-09-05T12:44:45 dbs++ okay, now I can do searches against our physical database server. "harry potter -insane" returned the appropriate results. 2008-09-05T12:44:49 hooray! 2008-09-05T12:44:53 yay! 2008-09-05T12:45:07 it took about 5-10 seconds to return that result. boo! 2008-09-05T12:45:24 sylvar: yeah, you're going to want to start playing with postgresql tuning now 2008-09-05T12:45:50 "funny" thing: I already have, a bit. 2008-09-05T12:46:53 shared_buffers = 2048MB, temp_buffers = 128MB, max_prepared_transactions = 30 2008-09-05T12:47:16 kernel.shmmax=2200000000, and kernel.shmall=2097152 2008-09-05T12:50:09 estimated_cache_size? warmed up filesystem cache? 2008-09-05T12:50:10 hmm, and when I play with work_mem and max_fsm_pages and restart postgres, I'm getting errors from the opac. (but not from postgresql-8.2 start) 2008-09-05T12:50:38 you need to restart services after restarting the db 2008-09-05T12:50:38 uh, restarting postgresql probably upsets the perl and c services 2008-09-05T12:50:54 ah 2008-09-05T12:51:29 sylvar: http://coffeecode.net/archives/165-Heating-up-Evergreen-search.html has some notes, although it's far (FAR) from the be-all end-all 2008-09-05T12:54:51 bbiab - lunch 2008-09-05T12:54:59 HOLY MOLY, MUCH BETTER 2008-09-05T12:55:07 never ever seen it work this fast on our system. 2008-09-05T12:55:30 for i in `/who #openils-evergreen` do; $i++; done 2008-09-05T12:56:05 * sylvar rejoices 2008-09-05T13:07:29 dbs: as for filesystem caching, we've only got 3GB of RAM on this server; it was cobbled together from available spare parts. 2008-09-05T13:31:10 memory is the cheapest resource in the known universe right now 2008-09-05T13:31:15 except for drive space 2008-09-05T13:31:57 (he said, pondering upgrading a machine of his own) 2008-09-05T13:46:03 *** kgs_writing is now known as kgs 2008-09-05T13:59:29 dbs: there? 2008-09-05T14:00:15 kbeswick_: nope 2008-09-05T14:00:29 DOH! You get me every time with that. Yep, I'm here. 2008-09-05T14:00:46 *** mjg_ has joined #openils-evergreen 2008-09-05T14:01:39 dbs: i need to figure out what i am doing next/what there is left to be done for autotools 2008-09-05T14:03:25 dbs: i am trying getting back into the swing of things with EG.. have been pretty busy this past week 2008-09-05T14:04:14 moi aussi 2008-09-05T14:04:50 kbeswick_: I would say, tackle the removal of the --with-db* options from configure.ac and move them into a separate utility for now 2008-09-05T14:05:46 dbs: add to an existing script , or make a new one? 2008-09-05T14:06:19 sboyette: yes, memory is insanely cheap, but the motherboard on this cobbled-together server might not support a whole lot more than 3GB 2008-09-05T14:06:37 * sylvar is still away, honestly 2008-09-05T14:06:46 which means that a complete install would be 1. ./configure --foo 2. eg_db_config (or whatever) --update-config --create-schema --user user --hostname hostname --password password 2008-09-05T14:07:06 kbeswick_: create a new script, that if you issue "--create-schema" will go off and run build-db.sh 2008-09-05T14:07:52 but maybe before that, try recovering from the move of Evergreen stuff into the Open-ILS directory 2008-09-05T14:08:46 and take a look at how you're munging of the setup.py stuff; I'm pretty sure that broke my python install and I had to manually remove the packages from that 2008-09-05T14:09:08 dbs: for the new script, would you suggest just doing it in sh, or perl, or what? 2008-09-05T14:09:39 I would use Perl 2008-09-05T14:09:57 it would probably be a good starting point for me to learn perl 2008-09-05T14:10:01 "just doing it in sh" usually results in something that would be easier to do in Perl 2008-09-05T14:10:12 and it will give you an XML parser 2008-09-05T14:14:46 *** agJohn has joined #openils-evergreen 2008-09-05T14:19:32 dbs: do you mean the setup.py stuff in ILS or SRF? 2008-09-05T14:20:23 my bad -SRF 2008-09-05T14:20:30 dbs: thought so 2008-09-05T14:21:02 the packages substitution blowed up real good 2008-09-05T14:21:29 Searching oddness. Before I dig deeper, anyone know offhand if there's something simple we have messed up? 2008-09-05T14:21:31 We've got this coming back (from time to time) from a test system. Haven't got a pattern yet to report except that it's happening on fairly broad searches. E.g. "keyword:criminal justice" (on a substantial university database) gives this: 2008-09-05T14:21:33 Results 1 - 10 of about NaN (page 1 of NaN) 2008-09-05T14:21:34 Now, I'm thinking that NaN is indicative of an overflow somewhere. I'm getting this in the open-ils.search...log: 2008-09-05T14:21:36 Use of uninitialized value in numeric eq (==) at /openils/lib/perl5/OpenILS/Application/Search/Biblio.pm line 766. 2008-09-05T14:22:20 Search probably timed out 2008-09-05T14:22:38 try "keyword:criminal justice -blah" 2008-09-05T14:22:45 should return nice and snappy 2008-09-05T14:23:28 OK... And I should tell the prospective user what ;~\ 2008-09-05T14:23:37 That did work fine, BTW. 2008-09-05T14:24:16 I actually got that on the second attempt at the search (1st one did not return--spinner just spun; DB had finished the query)... 2008-09-05T14:25:00 OK, so now I take off the " - blah" and it comes bad all right. Is this a caching issue or what? 2008-09-05T14:25:43 yep - cached by memcached 2008-09-05T14:26:09 tell the prospective user that the database needs tuning and/or more RAM 2008-09-05T14:26:53 and longer term, we probably need to fix the search timeout so that we don't send a garbage result back to memcached 2008-09-05T14:27:25 Yah, I'd endorse that approach. 2008-09-05T14:27:27 How much RAM would you think would be appropriate for a 2.7 million bib DB? 2008-09-05T14:28:40 ( "...comes bad..." up there was supposed to be "...comes back..." ) 2008-09-05T14:29:24 You can add up the size of your DB indexes (my last post on coffeecode.net details this process for the full-text indexes) and that will be a good start for sizing the RAM, methinks 2008-09-05T14:29:45 I defer very heavily to miker_ on database sizing and tuning, though 2008-09-05T14:29:56 only my toenails are wet so far 2008-09-05T14:31:38 I'll check out the post, thanks! 2008-09-05T14:31:57 Oh, and I understand that miker_'s The Man. 2008-09-05T14:33:42 I think for our ~4 million bib DB, I'm going to recommend upgrading from 16GB to at least 32GB of RAM - but still have a few more months of testing and tuning to refine that 2008-09-05T14:44:18 <_bott_> agJohn: ...using the way-back-machine, NaN = Not a Number ...Javascript speak for you tried to do arithmetic on a non-number 2008-09-05T14:44:54 *** _dkyle1 has left #openils-evergreen 2008-09-05T14:47:20 for the curious, google chrome does not like dojo 2008-09-05T14:48:37 or possibly the EG code we're attempting to load via dojo 2008-09-05T14:53:12 _bott_: Oh, yeah. I recognized NaN. I remember when the IEEE standard for floating-point was still a draft and we were fiddling with stuff in the128KB Mac (ick!) API's that allowed access to the 80-bit intermediate results.... Way back is right! 2008-09-05T14:56:41 Trying to get a compiler (even one that compiled to pseudo-code) into 128K was one of those worthless exercises (fortunately, I was only doing testing & docs of the language APIs--not trying to figure out how to shoehorn the code into inadequate RAM). One of the funniest things I saw was the original Macs had a little icon they'd pop up when you needed to switch diskettes. Well, we finally... 2008-09-05T14:56:43 ...had some 512K Macs and one of my guys was sitting there with a really funny look on his face and I went over and looked and it had the switch diskette icon up. He says to me: "I never know what to do when it says this." Closer inspection revealed that the "label" of the "diskette" it wanted was "RAM Disk".... 2008-09-05T14:56:45 Oh. 2008-09-05T15:03:51 *** mjg_ has left #openils-evergreen 2008-09-05T15:09:50 dbs: That post about heating up the cache is excellent. I'm interested to see what the appropriate balance is between Postgres shared_buffers and keeping the RAM accessible for the OS to use for caching. Do you have any preliminary thoughts on that? miker_: Any rules of thumb that have worked for biggy-sized DBs? (Right now I've got 1.5GB configured as shared_buffers in postgres.conf and... 2008-09-05T15:09:51 ...it's working all right, but I haven't tried the cache-heating technique of dbs's. 2008-09-05T15:32:40 agJohn: sorry, just got back into the office - the pendulum in postgresql-performance threads seems to be swinging more towards shared_buffers rather than os caching since 8.1 2008-09-05T15:32:48 but that's something I want to measure :) 2008-09-05T15:33:27 berick: meh (dojo / chrome) - have you tried plain-jane dijit tests? 2008-09-05T15:36:04 dbs: no, not all that concerned 2008-09-05T15:37:47 dbs: If you set a lot of RAM as shared_buffers, then the question is, do you do a select * on the index tables to heat up the cache? I'll be interested to see how your experiments turn out (and I'll sure let you know if I find out anything useful). Getting slow performance on the initial searches is not good for prospective users' morale... 2008-09-05T15:46:26 *** bruwal has joined #OpenILS-Evergreen 2008-09-05T15:52:23 *** bruwal has quit IRC 2008-09-05T16:08:37 agJohn: yeah (just finished phone call) - I kind of figured that warming up a shared_buffers cache might be a matter of just using curl or wget to issue a bunch of canned searches against the catalog (as that will also warm up the authority indexes, etc) 2008-09-05T16:08:46 have it run through a dictionary :) 2008-09-05T16:10:14 More complicated, for sure. 2008-09-05T16:53:16 *** dbs has quit IRC 2008-09-05T17:34:35 *** skmurphy has quit IRC 2008-09-05T17:36:25 *** skmurphy has joined #openils-evergreen 2008-09-05T18:21:33 *** rsinger_ has joined #OpenILS-Evergreen 2008-09-05T18:38:06 *** rsinger has quit IRC 2008-09-05T18:53:44 *** kgs is now known as kgs_away 2008-09-05T21:32:49 *** kgs_away is now known as kgs_night 2008-09-05T21:44:56 *** kgs_night has quit IRC 2008-09-05T21:53:14 *** kgs has joined #openils-evergreen 2008-09-05T23:19:00 *** kgs has quit IRC