#dev | Logs for 2015-05-16

« return
[00:03:29] <Bytram> okay, that's just TOO wierd!
[00:03:57] <Bytram> the main page story which I reposted today? Looked like all '?' on the main page? Now looks fine when I refreshed the main page.
[00:04:01] <Bytram> !!!?!!
[00:06:05] <Bytram> NCommander: paulej72: ^^^^^ ping
[00:08:03] <NCommander> Bytram, I didn't do anything on lithium as of late. I'm wondering if you're browser/OS combination is causing the 500
[00:09:16] <NCommander> Bytram, did you see my comments on the rehash bug?
[00:09:28] <Bytram> I ran my tests while following along with LiveHTTPHeaders active; all comms which specified a charset had: charset:UTF-8
[00:09:37] <Bytram> yes
[00:10:02] <Bytram> hence my giving it another go when I saw things had (seemingly) straightened out the earlier stories I'd posted.
[00:10:33] <Bytram> Bytram, freshenup updates articles and comment counts (??) can you elaborate a bit
[00:11:09] <NCommander> Bytram, due to performance reasons, a lot of stuff that rehash needs isn't calculated in realtime, but as part of slashd
[00:11:15] <Bytram> nod nod
[00:12:42] <NCommander> that bug has been an absolute showstopped :(
[00:13:25] <Bytram> better that I found it rather than the community
[00:13:58] <Bytram> hmmm, how often is the 'Most Recent Journal Entries' slashbox updated?
[00:14:06] <NCommander> Bytram, every few hours I believe.
[00:14:20] * Bytram posted the same UTF-8 test data as a journal article
[00:14:55] <Bytram> it looked okay when I pasted it in and when I previewed it.
[00:15:15] <NCommander> Bytram, nothing should be running an update though on the data itself; possibly on the same row though
[00:15:22] <paulej72> we need our utf8 master
[00:15:22] <Bytram> hmmm
[00:15:27] <NCommander> Honestly
[00:15:33] <Bytram> here's my journal article: https://dev.soylentnews.org
[00:15:33] <aqu4> ^ "3Journal of martyb (76)"
[00:15:36] <NCommander> Maybe we should let the community take a crack at it
[00:15:39] <Bytram> looks fine from here.
[00:16:03] <NCommander> Bytram, I see a lot of questionmarks
[00:16:52] <NCommander> Bytram, http://imgur.com
[00:16:52] <aqu4> ^ "3 Imgur"
[00:17:09] <Bytram> strange, it looks correct to me! There are 34 'replacement chars' in the first dozen or so rows, which is to be expected...
[00:17:26] <NCommander> Bytram, so what I see in that screenshot is correct?
[00:17:27] <Bytram> they are part of C0 and C1 control chars which need to be handled specially in HTML
[00:18:02] <NCommander> Bytram, looking at the raw HTML output, raw characters are coming back
[00:18:20] <Bytram> except for those, and about another 40-50 'boxed number' chars, all the rest have different glyphs.
[00:18:25] <NCommander> Pasted in contents of UTF-8gen.000001-0007ff.txt and hit 'preview' button. All looked okay. Now clicking on 'Save' button.�_�_�_�_�_�_�_�_ _ _�_�_ _�_�_� �_�_�_�_�_�_�_�_�_�_�_�_�_�_�_ !_"_#_$_%_&_'_(_)_*_+_,_-_._/_0 1_2_3_4_5_6_7_8_9_:_;__?
[00:18:33] <Bytram> as it should
[00:18:43] <NCommander> I thought we handled part of unicode as hrefs
[00:18:46] <Bytram> how 'bout later on when you get past U+0100 ??
[00:19:02] <NCommander> _ژ_ڙ_ښ_ڛ_ڜ_ڝ_ڞ_ڟ_ڠ ڡ_ڢ_ڣ_ڤ_ڥ_ڦ_ڧ_ڨ_ک_ڪ_ګ_ڬ_ڭ_ڮ_گ_ڰ ڱ_ڲ_ڳ_ڴ_ڵ_ڶ_ڷ_ڸ_ڹ_ں_ڻ_ڼ_ڽ_ھ_ڿ_ۀ ہ_ۂ_ۃ_ۄ_ۅ_ۆ_ۇ_ۈ_ۉ_ۊ_ۋ_ی_ۍ_ێ_ۏ_ې ۑ_ے_ۓ_۔_ە_ۖ_ۗ_ۘ_ۙ_ۚ_ۛ_ۜ_۝_۞_۟_۠ ۡ_ۢ_ۣ_ۤ_ۥ_ۦ_ۧ_ۨ_۩_۪_۫_۬_ۭ_ۮ_ۯ_۰ ۱_۲_۳_۴_۵_۶_۷_۸_۹_ۺ_ۻ_ۼ_۽_۾_ۿ_܀ ܁_܂_܃_܄_܅_܆_܇_܈_܉_܊_܋_܌_܍_܎_܏_ܐ ܑ_ܒ_ܓ_ܔ_ܕ_ܖ_ܗ_ܘ_ܙ_ܚ_ܛ_ܜ_ܝ_ܞ_ܟ_ܠ 
[00:19:02] <NCommander> ܡ_ܢ_ܣ_ܤ_ܥ_ܦ_ܧ_ܨ_ܩ_ܪ_ܫ_ܬ_ܭ_ܮ_ܯ_ܰ ܱ_ܲ_ܳ_ܴ_ܵ_ܶ_ܷ_ܸ_ܹ_ܺ_ܻ_ܼ_ܽ_ܾ_ܿ_݀ ݁_݂_݃_݄_݅_݆_݇_݈_݉_݊_݋_݌_ݍ_ݎ_ݏ_ݐ ݑ_ݒ_ݓ_ݔ_ݕ_ݖ_ݗ_ݘ_ݙ_ݚ_ݛ_ݜ_ݝ_ݞ_ݟ_ݠ ݡ_ݢ_ݣ_ݤ_ݥ_ݦ_ݧ_ݨ_ݩ_ݪ_ݫ_ݬ_ݭ_ݮ_ݯ_ݰ ݱ_ݲ_ݳ_ݴ_ݵ_ݶ_ݷ_ݸ_ݹ_ݺ_ݻ_ݼ_ݽ_ݾ_ݿ_ހ ށ_ނ_ރ_ބ_ޅ_ކ_އ_ވ_މ_ފ_ދ_ތ_ލ_ގ_ޏ_ސ ޑ_ޒ_ޓ_ޔ_ޕ_ޖ_ޗ_ޘ_ޙ_ޚ_ޛ_ޜ_ޝ_ޞ_ޟ_ޠ ޡ_ޢ_ޣ_ޤ_ޥ_ަ_ާ_ި_ީ_ު_
[00:19:03] <NCommander> ޫ_ެ_ޭ_ޮ_ޯ_ް ޱ_޲_޳_޴_޵_޶_޷_޸_޹_޺_޻_޼_޽_޾_޿_߀ ߁_߂_߃_߄_߅_߆_߇_߈_߉_ߊ_ߋ_ߌ_ߍ_ߎ_ߏ_ߐ ߑ_ߒ_ߓ_ߔ_ߕ_ߖ_ߗ_ߘ_ߙ_ߚ_ߛ_ߜ_ߝ_ߞ_ߟ_ߠ ߡ_ߢ_ߣ_ߤ_ߥ_ߦ_ߧ_ߨ_ߩ_ߪ_߫_߬_߭_߮_߯_߰ ߱_߲_߳_ߴ_ߵ_߶_߷_߸_߹_ߺ_߻_߼_߽_߾_߿_
[00:19:07] <NCommander>
[00:19:09] <NCommander> </div>
[00:19:13] <NCommander> I don't see a single href in the output
[00:19:21] <Bytram> nor should you!
[00:19:26] <NCommander> Ok, so this is correct?
[00:19:30] <Bytram> yes!!!
[00:19:34] <NCommander> THen ... WTF?
[00:19:48] <NCommander> Nothing beside taking lithium's NDB database offline should have affected this
[00:20:11] <Bytram> the basic idea is that whatever comes in from the user, inasmuchaspossible, should go into the DB, and what is in the DB should come back out to the user -- all without translation.
[00:20:28] <NCommander> Bytram, can you still break comments and/or articles?
[00:20:34] * NCommander can dump the raw table and see whats being stored
[00:20:47] <Bytram> there are a few exceptions, which escape me atm, but that's the gist of it as I understand it
[00:20:49] <NCommander> Its possible we've been misblaming perl on this
[00:21:02] <NCommander> The underlying storage engine for the site *was* changed
[00:21:10] <Bytram> when?
[00:21:15] <NCommander> Bytram, when I coded rehash
[00:21:34] <Bytram> yes, ISTR that.
[00:21:35] <NCommander> I didn't want to deal witha single point of failure, and slave failover would be a nightmare to have done in slash
[00:21:47] <NCommander> So I changed the underlying MySQL instance on dev to MySQL cluster
[00:22:03] <Bytram> k, go on
[00:22:22] <NCommander> slashcode used MySQL's InnoDB storage types except for the comments and story tables which were MyISAM
[00:22:33] <Bytram> blech!
[00:22:37] <NCommander> That's because the old search code used FULLTEXT() searches
[00:22:53] <Bytram> I guess, at the time, that was understandable.
[00:22:57] <NCommander> When I changed the database backend, everything became NDBENGINE, and I rewrote the search engine to use sphinix
[00:23:10] <Bytram> would have thought they'd have updated that after a while... but if it's working, don't fix it. eh?
[00:23:21] <Bytram> nod nod
[00:23:53] <NCommander> Bytram, I doubt the search code has been updated since 2005. Firehouse used sphinix for indexing, but I firebombed that code a year ago
[00:23:53] <Bytram> remember, this was all started with a single box in a guy's dorm room!
[00:24:16] <NCommander> Bytram, funny enough, SoylentNews (the site) started with a guy in an apartment in Panama, then added a lot of people :)
[00:25:05] <Bytram> my memory is fuzzy, but I think TMB had to make some changes in data.pm(?) to replace invalid chars with the 'replacement char' which is the blackdiamond-that-contains-a-question-mark
[00:25:12] <Bytram> ROFL!
[00:25:19] <NCommander> Bytram, yeah, and that's where perl craps out and we 500
[00:25:27] <NCommander> Bytram, but I can't reproduce the failure state
[00:25:44] <Bytram> hmm, A MAN A PLAN A ... ;)
[00:25:57] <NCommander> "The right man in the wrong place can make all the difference in the world ..."
[00:26:03] <Bytram> well, things ARE much better! I don't get 500's anymore
[00:26:23] * NCommander says fuck it, and starts a new branch on rehash
[00:26:29] <NCommander> UI translation is a go-go-go!
[00:27:24] <Bytram> that reminds me, I've not tried any non-ascii chars in story *titles* and a bunch of other places; so far it's only been in text-block entry fields.
[00:28:04] <NCommander> Bytram, if we can close the bug, I'm happy, but I dunno what fixed it
[00:28:09] <NCommander> It MIGHT return when we push to production
[00:28:21] <Bytram> well, it's not yet 'fixed'
[00:29:01] <Bytram> as I said, the first time I post a story to the main page, everything is fine ... up to the point that it first appears on the main page... then all chars > U+0100 appear as '?'
[00:29:12] <Bytram> and then, after some period of time, they suddenly look oaky!
[00:29:21] <Bytram> s/oaky/okay/
[00:32:30] <Bytram> NCommander: I just emailed you a copy of my test file: UTF-8gen.000001-0007ff.txt at your mc..@soylentnews.org email addy
[00:32:34] <NCommander> Bytram, got it
[00:33:15] <Bytram> I generally open the file in Emacs, select the whole file, copy it to the clipboard, go to Pale Moon, and paste it into the story text entry box.
[00:33:21] <NCommander> Bytram, your home page post looks right
[00:33:45] <Bytram> *NOW* it does, but not when I first posted it! :/ let me try it again and you'll see.
[00:33:59] * Bytram clicks submit story
[00:34:22] * Bytram enters title
[00:34:31] * Bytram selects topic: rehash
[00:34:41] <NCommander> Bytram, paulej72: WTF
[00:34:47] <NCommander> I ran freshenup by hand ... and it fixed it?!
[00:35:01] * NCommander feels his brain leak out
[00:35:06] <Bytram> ok... DO NOT DO ANYTHING!
[00:35:46] <Bytram> ouch.
[00:35:56] * Bytram fat fingered the wrong key and closed hexchatr
[00:36:03] <Bytram> I just pasted in UTF-8gen.000001-0007ff.txt
[00:36:10] <NCommander> Bytram, let me kill slashd
[00:36:14] <Bytram> format is Plain Old Text
[00:36:22] * Bytram clikcs "Preview Story"
[00:36:23] <NCommander> Bytram, ok, slashd is disabled
[00:36:28] <NCommander> It won't run freshenup until I do it manually
[00:36:30] <Bytram> k
[00:36:49] <Bytram> preview looks as it should; I see all sorts of interesting glyphs
[00:37:00] * Bytram clikcs SubmitStory
[00:37:26] * Bytram opens the story submission queue: https://dev.soylentnews.org
[00:37:31] <NCommander> hrm
[00:37:38] * Bytram opens the just-suibmitted sotry
[00:37:49] <Bytram> and all looks fine
[00:37:59] * Bytram clicks preview button
[00:38:05] <Bytram> stills looks fine
[00:38:19] <NCommander> Bytram, your post looks fine
[00:38:22] <NCommander> Mine looked fucked up now
[00:38:34] <NCommander> but when I reload its OK ...
[00:38:50] * Bytram sets release time to 2015-05-16 @ 00:38:00
[00:38:57] * Bytram clicks save
[00:39:00] <NCommander> Varnish or memcache
[00:39:05] <NCommander> Only two places this could be fucking up
[00:39:16] <Bytram> story is in queue at: https://dev.soylentnews.org
[00:39:16] <aqu4> ^ "320150515b - UTF-8gen.000001-0007ff.txt - POT - Dev.SN"
[00:39:18] <NCommander> Bytram, it looks good here
[00:39:23] <Bytram> looks fine, there
[00:39:44] * Bytram goes to tab containing: https://dev.soylentnews.org and hits Ctrl+F5 in Pale Moon
[00:40:10] <Bytram> and... it looks fine!?!
[00:40:24] <Bytram> https://dev.soylentnews.org
[00:40:24] <aqu4> ^ "320150515b - UTF-8gen.000001-0007ff.txt - POT - Dev.SN"
[00:40:27] <NCommander> Bytram, it also looks fine when I look at it via Apache directly
[00:40:40] <Bytram> okay....
[00:40:43] <NCommander> Bytram, it appears when we first submit an article, it doesn't ALWAYS render correctly
[00:40:57] <NCommander> On a reload whatever magic didn't happen and it renders properly
[00:41:00] <Bytram> you said you stopped slash?
[00:41:03] <NCommander> Yes
[00:41:06] <Bytram> *slashd?
[00:41:13] <Bytram> what if you restart it?
[00:41:17] <NCommander> I can do that now
[00:41:18] <NCommander> Standby
[00:41:20] <Bytram> k
[00:41:31] <NCommander> slash@lithium:~/rehash$ sudo service slash start
[00:41:32] <NCommander> Starting slashd slash: ok PID = 20986
[00:41:43] * Bytram notes THIS story's title is: 20150515b - UTF-8gen.000001-0007ff.txt - POT
[00:41:51] <paulej72> what the hell is freshenup doing?
[00:42:02] <NCommander> paulej72, it has an shitton of crap releating to rendering pages
[00:42:07] <NCommander> Which I *thought* we put a bullet in
[00:42:18] <NCommander> slash@lithium:~/rehash$ bin/runtask freshenup
[00:42:19] <NCommander> Sat May 16 00:42:14 2015 runtask freshenup.pl begin
[00:42:19] <NCommander> Sat May 16 00:42:14 2015 runtask freshenup.pl end: 0 (ran in 0.02s)
[00:42:28] <NCommander> No breakage here
[00:42:47] * Bytram hits refresh
[00:42:53] <Bytram> still looks fine
[00:43:20] <NCommander> Sat May 16 00:43:12 2015 runtask freshenup.pl begin
[00:43:20] <NCommander> Sat May 16 00:43:12 2015 runtask Updating 15/05/16/0038257
[00:43:20] <NCommander> Sat May 16 00:43:13 2015 runtask article.pl virtual_user=slash ssi=yes sid='15/05/16/0038257' cchp='aKHZQywdhI' section='mainpage' bytes=15773
[00:43:20] <NCommander> Sat May 16 00:43:14 2015 runtask freshenup.pl updated 940 mainpage:15/05/16/0038257 (20150515b - UTF-8gen.000001-0007ff.txt - POT) stderr: 'Use of uninitialized value $dir in substitution (s///) at /srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/Utility/Anchor.pm line 484. Use of uninitialized value in numeric eq (==) at /srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/Utility/Anchor.pm line 507. Use of uninitializ
[00:43:23] <NCommander> ed value in concatenation (.) or string at /srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/Utility/Anchor.pm line 507. Use of uninitialized value $dir in substitution (s///) at /srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/Utility/Anchor.pm line 484. Use of uninitialized value in numeric eq (==) at /srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/Utility/Anchor.pm line 507. Use of uninitialized value i
[00:43:28] <NCommander> n concatenation (.) or string at /srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/Utility/Anchor.pm line 507. '
[00:43:31] <NCommander> I just ran freshenup after a comment
[00:44:22] <Bytram> which comment?
[00:44:31] <Bytram> oh, nvm
[00:44:40] <Bytram> this one:
[00:44:41] <Bytram> https://dev.soylentnews.org
[00:44:42] <aqu4> ^ "3Dev.SN Comments | 20150515b - UTF-8gen.000001-0007ff.txt - POT"
[00:45:54] <Bytram> I just submitted that file UTD_8den.000001-0007ff.txt as a comment, without preview
[00:46:08] <Bytram> looks okay
[00:46:57] <NCommander> I'm hugely confused
[00:48:49] <Bytram> In all my other, prior attempts, ISTR that I initially saw '?' for all chars > U+0100 on initial appearance on the main page -- this is the first time I recall seeing it otherwise,
[00:49:03] <Bytram> I'm gonna try it again, same stuff, but without commentary as I go.
[00:49:14] <NCommander> Its *possible* taking lithium offline changed things
[00:49:22] <Bytram> could you please turn off slashd and refresh and whatever else you think might be messing with it?
[00:49:22] <NCommander> lithium's NDB database was having issues and kept OOMing
[00:49:35] <NCommander> Bytram, let me fire up Lithium's NDB database, and turn off slashd
[00:49:40] * Bytram likes chakras Oooooooooooooooooommmmmmmmm!
[00:49:41] <NCommander> This would give you the configuration you were testing on
[00:49:52] <Bytram> let me know when things are ready
[00:50:18] <NCommander> WTH
[00:50:26] <Bytram> ?
[00:50:26] <NCommander> NDB appears to be doing a rebuild
[00:50:34] <Bytram> wwhoah!
[00:50:36] <NCommander> Took forever to startup
[00:50:36] <NCommander> slash@lithium:~/rehash$ sudo -u mysql /opt/mysql/bin/ndbd
[00:50:36] <NCommander> 2015-05-16 00:50:10 [ndbd] INFO -- Angel connected to 'ipv4.helium.li694-22:1186'
[00:50:36] <NCommander> 2015-05-16 00:50:25 [ndbd] INFO -- Angel allocated nodeid: 3
[00:50:49] <NCommander> ndbd only been offline for a few hours
[00:50:56] <NCommander> Shouldn't have taken 15 seconds to start
[00:51:03] <NCommander> Bytram, test it now
[00:51:06] <Bytram> k
[00:51:19] <Bytram> I'll let ya know when I'm done; should be about a minute or so
[00:53:29] <Bytram> story saved and it looked okay all the way along.
[00:53:35] <Bytram> it looks okay on the main page, too.
[00:53:41] <Bytram> ???!!?!
[00:54:14] <Bytram> could it be that I'm setting the story 'go-live' time not far enough in the future to follow the right code path?
[00:54:24] * Bytram tries again
[00:54:26] <NCommander> Bytram, no idea
[00:54:33] <Bytram> give me another minute
[00:54:34] <NCommander> Bytram, I know TheMightyBuzzard might have fixed some unicode issues
[00:55:29] <paulej72> my.cnf was not set to do utf8. this was one of the major issues
[00:55:46] <paulej72> at least that is what TheMightyBuzzard said
[00:55:48] <NCommander> Interesting
[00:55:51] <NCommander> That might have fixed it
[00:56:05] * NCommander *thinks* he figured out how use gettext with i18n
[00:56:13] <Bytram> yes, but I saw the '?' appear *today*
[00:56:31] <paulej72> Bytram: no idea
[00:57:21] <NCommander> Bytram, TheMightyBuzzard may not have restarted NDBD when he changed my.cnf
[00:57:24] <Bytram> I set the story to go live at 01:01:00 UTC
[00:57:36] <NCommander> The other NDB instance is on helium which was historically set correctly
[00:57:54] <NCommander> When I killed NDBd on lithium, helium would have been the only node active.
[00:58:05] <Bytram> hrmm, but why did I see the '?' today... wait, did you say you restarted NDBD just a short while ago?
[00:58:18] <NCommander> Bytram, around 4-5ish?
[00:58:38] <Bytram> I don't think that would have done it, then.
[00:58:38] <NCommander> TheMightyBuzzard said he isn't familiar with NDBD, he might have restarted mysqld on lithium, but you also need to reboot the backend
[00:59:00] <NCommander> Bytram, I killed the one on lithium, helium has been up, but helium was the old DB master before we had neon and we had working UTF-8 then
[00:59:06] <NCommander> It'
[00:59:11] <NCommander> It's my.cnf would have been correct
[00:59:30] <NCommander> w/ lithium offline, the database would have failed over
[00:59:34] <Bytram> another minute or two and the story should appear
[00:59:58] <Bytram> and so, now, we'd have the right conf in place?
[01:00:05] <NCommander> Appears so
[01:00:16] <NCommander> paulej72, was this the last major rehash bug?
[01:00:42] <paulej72> search sucks balls
[01:00:48] <NCommander> paulej72, yeah but it did before.
[01:01:16] <NCommander> I mostly modified the code to use sphinix to reproduce what it was.
[01:01:17] <paulej72> but it seems like it sucks more.
[01:01:18] * Bytram loads dev
[01:01:31] <Bytram> looks good from here
[01:01:32] <NCommander> paulej72, I don't think I installed the indexer crontab.
[01:01:39] <NCommander> Which would mean its using old indexes
[01:01:41] <paulej72> I can't get it to sort most recent first
[01:01:52] <Bytram> stand on your head?
[01:02:11] <paulej72> crontab missing would do it
[01:02:51] <NCommander> paulej72, its running now
[01:02:59] <NCommander> slash@lithium:~/sphinx$ bin/indexer --all --rotate -c ../src/rehash/plugins/Search/sphinx.conf
[01:02:59] <NCommander> Sphinx 2.2.8-id64-release (rel22-r4942)
[01:02:59] <NCommander> Copyright (c) 2001-2015, Andrew Aksyonoff
[01:02:59] <NCommander> Copyright (c) 2008-2015, Sphinx Technologies Inc (http://sphinxsearch.com)
[01:03:00] <NCommander> using config file '../src/rehash/plugins/Search/sphinx.conf'...
[01:03:00] <NCommander> WARNING: key 'max_matches' was permanently removed from Sphinx configuration. Refer to documentation for details.
[01:03:03] <NCommander> indexing index 'rehash_stories_index'...
[01:03:06] <NCommander> paulej72, try it now
[01:03:22] <NCommander> it should suck less
[01:03:37] <Bytram> remove an 's' and you get suckles
[01:04:03] <NCommander> I'm worried this long dev cycle is killing momentium :(
[01:04:22] <Bytram> of whom? Us or the community?
[01:04:36] <Bytram> they're prolly just happy the system has been staying up!
[01:05:00] <NCommander> us
[01:05:17] * Bytram realizes that he just tempted fate with that remark and a big sys upgrade right ahead.
[01:05:24] <paulej72> NCommander: I will need to try some of the code I was working on previuosly, but right now it just displays the oldest first
[01:06:00] * NCommander is fiddling with a branch of rehash
[01:06:48] * Bytram watches a fledging drop from its nest and commence its first flight
[01:07:03] <Bytram> afk break brb
[01:08:55] * NCommander fiddles with xgettext.pl
[01:12:05] <paulej72> Bytram: can you post a pict of the expected output for the current utf8 test
[01:14:08] <Bytram> back
[01:14:14] <Bytram> I just remembered something...
[01:14:26] <Bytram> there's multiple other places where story text, etc. comes out...
[01:14:41] <Bytram> e-mail summary of day's stories
[01:14:44] <Bytram> NNTP
[01:14:46] <Bytram> RSS
[01:14:51] <Bytram> other?
[01:15:03] <Bytram> paulej72: do we have a place I can post it to?
[01:16:25] <paulej72> pastebin
[01:16:58] <paulej72> wait, how about intagram
[01:17:00] <Bytram> ugh. I try to use outside stuff as much as possible
[01:17:08] <Bytram> never used either of them
[01:17:16] <Bytram> don't WE have a place I can post it to?
[01:17:29] <paulej72> i am thiniking
[01:17:50] <paulej72> wiki
[01:19:19] <NCommander> slash@lithium:~/src/rehash$ ~/perl/bin/xgettext.pl -P tt2 -g -u -v -w themes/default/templates/admin\;menu\;default
[01:19:19] <NCommander> * themes/default/templates/admin;menu;default
[01:19:19] <NCommander> - Total strings extracted : 7
[01:19:19] <NCommander> READING PO FILE : messages.po
[01:19:20] <NCommander> WRITING PO FILE : ./messages.po
[01:19:20] <NCommander> DONE
[01:19:22] <NCommander> WOOHOO
[01:19:22] <Bytram> interesting... How would I post a file there?
[01:19:26] * NCommander figured out how to get TT2 to do gettext
[01:20:04] <Bytram> noice!
[01:20:06] <NCommander> paulej72, Bytram: upload it as a file to an article
[01:20:36] <Bytram> hasn't seemed to work before... maybe I didn't do it right? that was a LONG time ago, though.
[01:20:41] <NCommander> Bytram, its been fixed
[01:20:46] <paulej72> Bytram: edit a file and there is an icon to upload a file
[01:20:50] <Bytram> =)
[01:20:51] <NCommander> I think I fixed it a few releases ago
[01:21:27] * NCommander starts extracting strings in our templates to build a POT
[01:22:20] * Bytram is trying it now
[01:22:40] * Bytram selected the file, hit preview and got:
[01:22:40] <Bytram> Request Entity Too Large
[01:22:40] <Bytram> The requested resource
[01:22:40] <Bytram> does not allow request data with POST requests, or the amount of data provided in the request exceeds the capacity limit.
[01:23:06] <Bytram> was only 109,918 bytes
[01:23:14] <Bytram> paulej72: why don't I just email it to you?
[01:23:22] <paulej72> sure
[01:23:50] <Bytram> I'll use: paulej72@soylentnews.org
[01:23:52] <paulej72> wonder if github will take it as part of the issue comment
[01:24:02] <paulej72> should work for email
[01:25:37] <NCommander> Bytram, oh thats a rehash bug. I need to increase the capacity limit in httpd.conf
[01:25:57] <Bytram> =) HTH!
[01:26:05] <Bytram> paulej72: should have it, now
[01:26:32] * Bytram goes to check someything
[01:27:27] <NCommander> #: themes/default/templates/about;about;default:26
[01:27:27] <NCommander> msgid "Frequently Asked Questions"
[01:27:27] <NCommander> msgstr ""
[01:27:28] <NCommander> Neat
[01:30:11] <Bytram> NCommander: paulej72: In case you'd like some background, take a look at: http://soylentnews.org and scroll down to the part in bold: "A little more background:" which lists the chars that should be replaced with the black-diamond-around-a-?-char, i.e. the 'Replacement Character'
[01:30:11] <aqu4> ^ "3Journal of martyb (76)"
[01:30:58] <Bytram> so far, I've only done tests with 1-octet and 2-octet UTF-8 chars... I should try 3- and 4-octet chars, too.
[01:37:32] <Bytram> from: https://tools.ietf.org
[01:37:32] <aqu4> ^ "3403 Forbidden"
[01:37:32] <Bytram> #
[01:37:32] <Bytram> # The table below summarizes the format of these different octet types.
[01:37:32] <Bytram> # The letter x indicates bits available for encoding bits of the
[01:37:32] <Bytram> # character number.
[01:37:33] <Bytram> #
[01:37:34] <Bytram> # Char. number range | UTF-8 octet sequence
[01:37:36] <Bytram> # (hexadecimal) | (binary)
[01:37:38] <Bytram> # --------------------+---------------------------------------------
[01:37:40] <Bytram> # 0000 0000-0000 007F | 0xxxxxxx
[01:37:42] <Bytram> # 0000 0080-0000 07FF | 110xxxxx 10xxxxxx
[01:37:44] <Bytram> # 0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx
[01:37:46] <Bytram> # 0001 0000-0010 FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
[01:37:48] <Bytram> #
[01:37:50] <Bytram> LOL
[01:42:04] <NCommander> This is tedious :(
[01:42:12] * NCommander is going to have a full set of rehash templates made
[01:42:18] <NCommander> I want SN to go beyond English
[01:44:11] <Bytram> NCommander: do you remember the story we ran about a long-standing bug about a single Polish letter that caused a bug in a web site?
[01:44:38] <Bytram> there was a link in that story that was REALLY eye-opening for me!
[01:45:04] <NCommander> Bytram, about i10n?
[01:45:16] * NCommander is going to get rehash localizable
[01:46:36] <Bytram> the most striking thing I got from one of the linked stories was problems in counting and enumeration ...
[01:47:11] <Bytram> the example escapes me, but it basically precluded something like: "choose among these $n options"
[01:47:50] <Bytram> NCommander: can you please bump the allowable image size so I can send that pic to pj? his e-mail server is not accepting e-mail from SN
[01:50:21] * Bytram submits a story with 3-octet UTF-8 chrs
[01:52:01] * Bytram notes his installed font has lots of missing chars, but all has looked good through the submission process; now waiting for it to appear on dev's main page
[01:52:15] <Bytram> should appear at 01:52 UTC
[01:52:48] <NCommander> Bytram, paulej72: the start of rehash i18n
[01:52:56] <Bytram> and, there it is: "20150515e - UTF-8gen.000800-000fff.txt - POT" https://dev.soylentnews.org
[01:52:57] <aqu4> ^ "3Dev.SN: Dev.SN is dead developers"
[01:53:11] <NCommander> http://pastebin.com
[01:53:11] <aqu4> ^ "3# SOME DESCRIPTIVE TITLE. # Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER - Pastebin.com"
[01:53:15] <Bytram> NCommander: how much i18n stuff have you done before?
[01:53:34] <NCommander> Bytram, some
[01:53:40] * Bytram has done a fair amount, back in the double-wide charset days.
[01:53:45] <Bytram> nod nod
[01:53:53] <NCommander> Bytram, Launchpad was fully i18n-ed, and I had to write code that comformed to that, but I've never taken a massive legacy codebase and made it translatable
[01:54:19] <NCommander> Bytram, I know no other slashcode site tried *this*
[01:54:24] <Bytram> do you know how to tell if the text you are seeing displayed is something still imbedded in the code or if it is properly coming from the translation file?
[01:54:41] <NCommander> Bytram, I used to use a fake template with gibberihs in it
[01:54:49] <NCommander> WHenever I found an actual string, it meant I missed something :)
[01:54:59] <Bytram> I've got something better!
[01:55:24] <Bytram> "aBORT? rETRY? fAIL?"
[01:55:43] <Bytram> *inverse* capitalization of english text in the lookup file
[01:56:03] <Bytram> that way, you KNOW you got the right message AND that you got it from the right place!
[01:56:12] <Bytram> =)
[01:58:16] <Bytram> NCommander: did you bump the size of a file that I can paste into a story? ATM, I need at least 110,000 bytes, so if you could make it 250,000 that would be greata!
[01:58:26] <paulej72> use Klingon
[02:01:10] <NCommander> Bytram, I didn't
[02:01:16] <NCommander> paulej72, Launchpad Rosetta supports klingdon
[02:01:44] <Bytram> it's okay, am trying another route
[02:02:05] <NCommander> paulej72, https://translations.launchpad.net
[02:02:07] <aqu4> ^ "3Klingon (tlh) : Vivid (15.04) : Translations : Ubuntu"
[02:02:39] <NCommander> Bytram, it should work now
[02:02:45] <Bytram> NCommander: would be nice if we had a pastebin.soylentnews.org where we could pass files around; have a daemon that auto deleted all files more than, say, 12 hours old.
[02:03:14] * Bytram tries it again
[02:03:32] * Bytram has attached a file and hits preview on dev
[02:03:50] <Bytram> NCommander:
[02:03:50] <Bytram> OK
[02:03:51] <Bytram> The server encountered an internal error or misconfiguration and was unable to complete your request.
[02:03:51] <Bytram> Please contact the server administrator, slash@dev.soylentnews.org and inform them of the time the error occurred, and anything you might have done that may have caused the error.
[02:03:51] <Bytram> More information about this error may be available in the server error lo
[02:03:57] <NCommander> -_-;
[02:03:59] <paulej72> Bytram: the pict you sent is the expected output correct
[02:04:12] <NCommander> [Sat May 16 02:02:39 2015] [notice] Apache/2.2.29 (Unix) DAV/2 mod_apreq2-20090110/2.8.0 mod_perl/2.0.8 Perl/v5.20.1 configured -- resuming normal operations
[02:04:12] <NCommander> [Sat May 16 02:03:37 2015] [error] Can't call method "upload" on an undefined value at /srv/soylentnews.org/rehash/site/soylent-mainpage/htdocs/admin.pl line 1938.\n
[02:04:12] <NCommander> Use of uninitialized value in lc at /srv/soylentnews.org/rehash/site/soylent-mainpage/htdocs/comments.pl line 26.
[02:04:15] <NCommander> DA FUCK
[02:04:32] <Bytram> yes. as far as I understand it, everything I see there is explainable and makes sense to me.
[02:04:45] <NCommander> if ($form->{media_file}) {
[02:04:45] <NCommander> my $upload = $form->{query_apache}->upload;
[02:04:47] <NCommander> $!LJFIJHEOIHRFI#
[02:04:54] <NCommander> mod_perl 2 shit we didn't find
[02:05:18] <Bytram> well, *we* didn't find, but I seem to have done okay. ;)
[02:06:08] <NCommander> I have no idea how this is uspposed to work
[02:06:49] * NCommander is annoyed
[02:06:51] * Bytram wish I could help you out with that one.
[02:07:06] <Bytram> brb short break
[02:07:47] <NCommander> Investigating
[02:10:34] <NCommander> crap
[02:10:39] <NCommander> I don't remember how you summon $r
[02:12:25] <Bytram> start by drawing a pentagram?
[02:12:44] <paulej72> Bytram is correct
[02:13:11] * NCommander tests a fix
[02:13:55] <Bytram> You guys are the best! There's no place else I can joust with such fine minds!
[02:17:00] <Bytram> NCommander: I just sent you the file that I've been trying to upload as part of a story; sent to your SN addy
[02:17:16] <NCommander> Bytram, any file will do it
[02:17:19] <NCommander> Its an API break
[02:18:17] <Bytram> ahhh. got it. thanks!
[02:19:33] <NCommander> suprised we didn't catch this before
[02:19:40] * NCommander would love to get rehash certified for release
[02:23:36] <NCommander> ugh
[02:23:39] <NCommander> I remember how I fixed this
[02:23:39] <NCommander> # UNBELIEVE HACKINESS AHEAD
[02:23:39] <NCommander> #
[02:23:39] <NCommander> # Ok, under MP1, it was possible to use param as a "semi-persistant" scratchpad
[02:23:41] <NCommander> # that is, to save a new element in the hashref, and get it back by future calls
[02:23:41] <NCommander> #
[02:23:43] <NCommander> # This worked because the older APR methods allowed you to store into the HASHREF.
[02:23:44] <NCommander> # even though this behavior was wrong, and bad according to MP documentation. MP2
[02:23:48] <NCommander> # now removed the STORE method from the APR tables so any attempt to write to them
[02:23:50] <NCommander> # goes BANG.
[02:23:52] <NCommander> #
[02:23:54] <NCommander> # Since we can't do that now, we're going to have to fake it. On our first call to
[02:23:56] <NCommander> # getCurrentForm, we'll copy the param tables to a hashref, then shove it into the
[02:23:58] <NCommander> # apache2 pnotes, and then retrieve it on demand.
[02:24:00] <NCommander> #
[02:24:02] <NCommander> # This is a fucking hack, but I can't think of a better way than to refactor a TON of
[02:24:04] <NCommander> # perl, and perl is not a language that makes it easy to refactor ...
[02:24:57] <Bytram> thank $diety for copious comments
[02:26:59] <NCommander> APR::Request::Param::Table=HASH(0x75d1760)
[02:27:00] <NCommander> Well
[02:27:02] <NCommander> *cough*
[02:27:06] <NCommander> partial fix
[02:29:45] <Bytram> should I give it another try on dev?
[02:29:48] <paulej72> NCommander: you need to call it correctly
[02:30:49] <NCommander> paulej72, ... it appears that the upload() method was never ported over to MP2
[02:30:51] <NCommander> -_-;
[02:31:56] <paulej72> sucks then
[02:32:29] <NCommander> Or maybe it was
[02:32:31] <NCommander> Hard to tell
[02:34:16] <paulej72> Bad: my @thismany = $form->{$input};my @thismany = @{$form->{$input}};
[02:34:41] <paulej72> Good: my @thismany = @{$form->{$input}}
[02:34:58] <paulej72> line got messed on on the first
[02:35:51] <NCommander> paulej72, I actually need Apache2::Upload not to get that APR request
[02:35:52] <paulej72> got to reference the hash correctly or else you get the pointer like above. Wonder if that is the issue with:APR::Request::Param::Table=HASH(0x75d1760
[02:41:12] <NCommander> paulej72, I need the upload methods which are overloaded by Apache2->upload -_-;
[02:48:08] <NCommander> Ok
[02:48:10] <NCommander> got it fixed
[02:48:15] <NCommander> Now to fix ti everywhere
[02:48:18] <NCommander> What a fucking pain
[02:54:17] <NCommander> fixed. I think
[03:03:41] <NCommander> woohoo jackpot
[03:03:44] * NCommander gets a branch to merge
[03:06:09] <NCommander> blobs fixed
[03:06:12] <NCommander> fully now
[03:10:11] <crutchy> perl--
[03:10:11] <Bender> karma - perl: -1
[03:11:36] <NCommander> paulej72, TheMightyBuzzard https://github.com
[03:11:36] <aqu4> ^ "3Fix blob module to handle MP2 API stupidness by NCommander · Pull Request #44 · SoylentNews/rehash · GitHub"
[03:18:14] <NCommander> https://github.com
[03:18:15] <aqu4> ^ "3Blob fix by NCommander · Pull Request #45 · SoylentNews/rehash · GitHub"
[03:18:16] <NCommander> correction