#dev | Logs for 2015-06-02
« return
[01:02:10] -!- cmn32480 [cmn32480!~cmn32480@Soylent/Staff/Editor/cmn32480] has joined #dev
[03:44:29] -!- cmn32480 has quit [Quit: See You Later]
[04:15:00] -!- arti has quit [Quit: Ende gut, alles gut.]
[04:16:34] -!- arti [arti!~arti@iminylescdbcem.com] has joined #dev
[05:05:17] -!- arti has quit [Quit: Ende gut, alles gut.]
[05:05:49] -!- arti [arti!~arti@iminylescdbcem.com] has joined #dev
[05:23:26] -!- arti has quit [Quit: Ende gut, alles gut.]
[05:25:31] -!- arti [arti!~arti@iminylescdbcem.com] has joined #dev
[05:43:18] -!- artificial [artificial!arti@cwz-693-36-504-71.socal.res.rr.com] has joined #dev
[09:21:08] crutchy is now known as poutine--
[09:22:04] poutine-- is now known as crutchy
[11:17:23] -!- Tachyon has quit [Read error: Connection reset by peer]
[11:19:18] -!- Tachyon [Tachyon!~Tachyon@xuco.me] has joined #dev
[11:54:05] <CoolHand> anyone notice that the "hits" counter for the story list on admin.pl seems broken? most stories since the upgrade have n/a, and just about all the others show far less hits than comments...
[12:56:35] -!- TLA [TLA!~52027da6@bzd3-bxxi33-7-6-smos942.64-6.cable.virginm.net] has joined #dev
[12:57:05] <TLA> Paul you about?
[12:59:48] -!- janrinok [janrinok!~janrinok@Soylent/Staff/Editor/janrinok] has joined #dev
[12:59:50] <TLA> getting 500 internal server errors here, it appears to be entirely random within SN
[13:08:03] -!- TLA has quit [Quit: Web client closed]
[13:30:18] <CoolHand> I concur with TLA (and others have reported)
[13:34:23] <mrcoolbp> paulej72: they are getting "500 internal server errors"
[13:37:10] <paulej72> mrcoolbp: I have not seen them
[13:37:17] <mrcoolbp> neither have I
[13:37:28] <mrcoolbp> TLA and CoolHand mentioned it above
[13:37:29] <janrinok> I have - twice today
[13:37:33] <mrcoolbp> ^
[13:37:37] <paulej72> I need more info on just when they are happening
[13:38:26] <mrcoolbp> site is nice and snappy for me today
[13:39:48] <janrinok> OK, the two for me were 1. when I selected edit from a story on the front page (should only affect editors), and the second is on the lower half of the page when replying to a comment. The story prints OK, but the comment and the text box fail.
[13:40:52] <CoolHand> I had one when posting a comment and one when I clicked on "stories" link at top
[13:41:19] <CoolHand> F5 refresh immediately came back with proper pages
[13:42:19] <paulej72> were the 500's comming up immediatly or was there a delay?
[13:42:27] <janrinok> immediately
[13:43:09] <janrinok> ... well as quickly as the page displayed
[13:43:25] <janrinok> no undue delay from the usual
[13:45:11] <mrcoolbp> paulej72: Internal Server Error when trying to preview a comment
[13:45:40] <mrcoolbp> paulej72: also the error message says to contact " slash@dev.soylentnews.org "
[13:46:26] <paulej72> NCommander's fault for copying the data from dev instead of doing a fresh install
[13:46:48] <paulej72> apache config file is set with the wrong data
[13:54:35] <paulej72> OK updated apache config and restarted
[13:55:53] <janrinok> ah, that's why it just logged me out.....
[13:58:00] <CoolHand> hmm, didn't log me out... weird..
[13:58:54] <janrinok> I tried to update a page during reboot I guess
[14:51:42] -!- d33tah [d33tah!~d33tah@lczcb.math.uni.lodz.pl] has joined #dev
[14:52:29] <d33tah> hi, i'm getting the impression that after the site update, my rss entries got doubled in thunderbird. since "Wooden Chip: Biodegradable Semiconductors", every entry is displayed as double.
[15:00:58] <paulej72> d33tah: if your rss feed reader saves old rss data then you may have duplicates as there was a misconfg with the site that cuased the rss feed to go out with the wrong urls. your rss reader may have thought those two were different. I will probably do the same now that the feeds are fixed.
[15:08:52] <paulej72> anyone noticed any 500 errors since i restared apache?
[15:21:27] <janrinok> not been on the site very much - looking for new stuff - but nothing seen so far
[16:33:22] <CoolHand> haven't noticed any, but haven't been on too much either...
[17:10:29] * NCommander is awake again
[17:20:23] <paulej72> NCommander: I gotten most of the rehash bugs fixed
[17:24:40] <paulej72> NCommander: I am looking for the command to set varnish to only listen on port 80. do you know the answer
[17:26:02] <NCommander> paulej72, its a command line argument to varnish in /etc/defaults/varnish
[17:26:18] * NCommander feels horridly guilty dumping this much onto paulej72
[17:26:42] <paulej72> not a problem, the getSkins bug was my fault anyway
[17:27:09] <NCommander> paulej72, any issues if I wipe hydrogen and rebuild it?
[17:27:31] <paulej72> I want to test the varnish config on there first.
[17:27:37] <NCommander> ok
[17:27:42] <paulej72> it should only take a few minutes
[17:27:42] <NCommander> Let me know when we're good to nuke and pave
[17:32:43] <paulej72> NCommander: I am having trouble getting to port 2626 on prod. it should go to the ipn deamon but it does not. I can get to it locally with lynx, but not from the outside. any idea why. I thought varnish was the issue but it is not
[17:33:16] <paulej72> http://198.58.127.22:2626 should give a webpage that spits back the /test string for hydrogen
[17:33:31] <aqu4> Failed to connect to retrieve page title.
[17:35:03] <paulej72> ok ufw is running
[17:38:21] <NCommander> paulej72, is IPNd listening on IPv6
[17:38:32] <paulej72> no
[17:40:39] <NCommander> paulej72, do you have IPv6 connectivity?
[17:40:42] <NCommander> That might be the issue
[17:40:55] <paulej72> the port was not opened
[17:47:30] <paulej72> NCommander: I am done with hydrogen, but can you tell me why mysqld is running on hydrogen and flourine? I thought we only needed that on helium and neon.
[17:49:35] <paulej72> NCommander: is there any tuning that needs done to make mysql access faster?
[17:54:38] <janrinok> "Server Configuration Error" followed by 502 when trying to edit submission
[17:54:58] <janrinok> Works on 4th attempt - ???
[17:57:46] <paulej72> deployed bug fixes to site
[17:58:03] <janrinok> k thx
[17:58:12] <paulej72> as they were big ones, just did it without notification :)
[17:59:56] <paulej72> NCommander: we are still seeing delays on fluorine when loading stories. It almost seems like it might be db delays. is there a way to check
[18:05:02] <paulej72> no more _none :)
[18:39:35] <NCommander> paulej72, ndbd is running on neon and hydrogen which acts as the data store and a lot of the heavy lifting. mysqld is the frontend to ndbd in cluster
[18:40:18] * NCommander is really not focusing well today
[18:41:58] <paulej72> NCommander: there are bout 50 different mysqld process running on fluroine. that seems excessive
[18:43:21] <NCommander> paulej72, in htop or ps ax?
[18:43:44] <paulej72> htop, but they all have differnet pids
[18:45:46] <paulej72> ok looks like threads
[18:46:09] <paulej72> I thought I had treads truned off
[18:46:35] <NCommander> Node 2: Data usage is 72%(47531 32K pages of total 65536)
[18:46:35] <NCommander> Node 2: Index usage is 22%(14472 8K pages of total 65568)
[18:46:35] <NCommander> Node 3: Data usage is 72%(47528 32K pages of total 65536)
[18:46:35] <NCommander> Node 3: Index usage is 22%(14456 8K pages of total 65568)
[18:46:41] <NCommander> Caching in the cluster seems OK
[18:47:05] <paulej72> still seems like a lot snice there seems to be only 4 or 5 busy at the most.
[18:49:10] <NCommander> paulej72, 4-5 busy?
[18:49:17] * NCommander is looking at the tuning parameters
[18:50:57] * NCommander does a rolling restart of the cluster
[18:52:37] <paulej72> well it is fun looking at the spikey nature of mysqld on fluroine.
[18:52:49] -!- janrinok has quit [Quit: byeeeee]
[18:56:54] <NCommander> paulej72, I reworked the caches
[18:57:45] <NCommander> paulej72, trying to figure out if the site is less slow
[18:58:01] <paulej72> looking myself as well
[18:58:26] <paulej72> seems about the same
[18:59:31] <paulej72> do we need to tune mysqld on the frontends
[18:59:44] <NCommander> paulej72, possibly
[19:11:02] -!- Tachyon has quit [Read error: Connection reset by peer]
[19:11:03] -!- Tachyon_ [Tachyon_!~Tachyon@xuco.me] has joined #dev
[19:17:17] <NCommander> paulej72, I'm setting up slow query logging on fluorine's frontend
[19:17:28] <NCommander> Let's see if we can figure out where we're excessively hitting the database
[19:17:35] <paulej72> k
[19:20:29] <NCommander> paulej72, w00t
[19:20:36] <paulej72> yes
[19:20:39] <NCommander> paydirt, we're getting a list of queries thats taking more than 3-4 seconds
[19:20:56] <paulej72> yes are they ones I made?
[19:21:08] <paulej72> where is the list?
[19:21:11] <NCommander> The primary offender seems to be SELECT stories.stoid, sid, title, stories.tid FROM story_text, story_topics_rendered,
[19:21:11] <NCommander> stories LEFT JOIN story_param ON (stories.stoid=story_param.stoid
[19:21:11] <NCommander> AND story_param.name='neverdisplay' AND story_param.value != 0) WHERE stories.stoid = story_text.stoid
[19:21:11] <NCommander> AND stories.stoid = story_topics_rendered.stoid
[19:21:14] <NCommander> AND '2015-05-01 04:46:00' > DATE_SUB('2015-06-02 19:20:00', INTERVAL 120 DAY)
[19:21:14] <NCommander> AND time < '2015-05-01 04:46:00'
[19:21:16] <NCommander> AND time <= '2015-06-02 19:20:00'
[19:21:20] <NCommander> AND in_trash = 'no'
[19:21:22] <NCommander> AND story_param.stoid IS NULL
[19:21:24] <NCommander> AND story_topics_rendered.tid = 1 AND stories.stoid != '5986' GROUP BY stories.stoid ORDER BY time DESC LIMIT 1;
[19:21:27] <NCommander> paulej72, /var/log/mysql/slow-query.log
[19:21:35] <NCommander> there's a postprocessor which summarizes the file after we let it run for awhile
[19:22:48] * NCommander can do some basic DBAing to get better performance
[19:24:11] <NCommander> But in short, its the story load thats killing our performance
[19:24:21] <paulej72> that where clause is crazy. it seems it is doing the same thing multiple times
[19:28:26] <NCommander> Yeah, and its done for *every* page load
[19:31:07] <NCommander> paulej72, I'm hoping I can fix this with some additional indexes
[19:31:12] <NCommander> Else refactoring is necessary
[19:31:17] <paulej72> k
[19:32:42] <paulej72> NCommander: what is concidered a long query anything over 3 seconds? can we look at stuff in the 1 to 2 second range
[19:32:57] <NCommander> paulej72, I'll set it to 1 second
[19:33:14] <NCommander> paulej72, the variable was set to 2
[19:33:42] <NCommander> +----+-------------+-----------------------+--------+-----------------------+-----------+---------+-----------------------------------------------+------+------------------------------------------------------------------------------------+
[19:33:43] <NCommander> | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
[19:33:43] <NCommander> +----+-------------+-----------------------+--------+-----------------------+-----------+---------+-----------------------------------------------+------+------------------------------------------------------------------------------------+
[19:33:45] <NCommander> | 1 | SIMPLE | story_topics_rendered | range | story_topic,tid_stoid | tid_stoid | 7 | NULL | 6244 | Parent of 3 pushed join@1; Using where; Using MRR; Using temporary; Using filesort |
[19:33:51] <NCommander> | 1 | SIMPLE | stories | eq_ref | PRIMARY,time | PRIMARY | 3 | soylentnews.story_topics_rendered.stoid | 1 | Child of 'story_topics_rendered' in pushed join@1; Using where |
[19:33:55] <NCommander> | 1 | SIMPLE | story_text | eq_ref | PRIMARY | PRIMARY | 3 | soylentnews.story_topics_rendered.stoid | 1 | Child of 'stories' in pushed join@1 |
[19:33:59] <NCommander> | 1 | SIMPLE | story_param | eq_ref | story_key | story_key | 101 | soylentnews.story_topics_rendered.stoid,const | 1 | Using where; Not exists |
[19:34:03] <NCommander> +----+-------------+-----------------------+--------+-----------------------+-----------+---------+-----------------------------------------------+------+------------------------------------------------------------------------------------+
[19:34:07] <NCommander> eek
[19:34:08] <paulej72> The current slow queries seem like the ones slashd uses to get stories to rebuild
[19:34:09] <NCommander> That's pretty horrific explain
[19:34:11] <NCommander> basically its looping 6244 times
[19:34:34] * NCommander tries to find this monster in rehash
[19:35:37] <NCommander> paulej72, sub getStoriesByTime in Slash/DB/MySQL/MySQL.pm
[19:35:54] <paulej72> wow that was quick
[19:36:03] <NCommander> I'm good with grep ;)
[19:36:30] <NCommander> paulej72, that's called by article.pl
[19:37:32] <paulej72> getStoryByTime?
[19:38:48] <NCommander> paulej72, yeah
[19:39:09] <NCommander> Everytime getStoryByTime, httpd hangs waiting for cluster to spit back the rows
[19:39:29] <NCommander> Since we're still using prefork for compatbility reasons, we're limited on the number of things Apache can handle at once
[19:39:57] <NCommander> Since article.pl hangs up httpd, more than a few hits at a time, httpd starts pending connections until other stuff becomes available
[19:41:26] <paulej72> NCommander: getStoryByTime is fubar'd with repect to nexus support. It does not stay within the bounds of the current nexus if not mainpage
[19:42:27] <paulej72> This was a bug I found wen debugging stuff yesterday, but I was not sure where it came from, jsut noticed that pervious and next articles were not based on the nexus for meta and breeaking
[19:42:44] <NCommander> Exciting :(
[19:43:01] <NCommander> It looks like indexes are in place where they need to be
[19:43:15] <NCommander> This query was probably bad pre-cluster, and just made worse due to the distributed nature of cluster
[19:44:16] <paulej72> yep, I did check my monster one for mod bombs, and it did not trigger an issue
[19:45:18] <paulej72> if we can get rid of the two time < x clauses that would be better
[19:45:47] <NCommander> I'm looking at it
[19:46:02] <NCommander> Its probably faster to return more data and get perl to exclude what we don't want
[19:47:49] <paulej72> probably faster to make the where clause much simpler by using some perl date manipulations. We already have the perl modules loaded as I used dat manips for subscriptions
[19:48:54] <NCommander> paulej72, dropping the time part doesn't speed up the query
[19:57:53] <paulej72> who decided to put the neverdisplay for a story as a pram instead of a colum in the story table
[19:58:18] <NCommander> paulej72, the original devs :)
[19:58:26] * NCommander is trying to make heads or tails of this crap
[19:58:53] <NCommander> Ok
[19:58:54] <NCommander> Ew
[19:59:07] <paulej72> i want to know who as that is what is killing performance. Having to look in that table to check for never display si a big hit
[19:59:35] <NCommander> paulej72, ok, I get what's happening
[19:59:49] <NCommander> When in cluster most, JOIN queries are normally load balanced across the system, and run on the backend itself
[19:59:53] <NCommander> *mode
[20:00:09] <NCommander> That breaks if a BLOB-type table is in use, in this case text
[20:00:23] <NCommander> Which then requires mysqld to pull data bit by bit via the NDB backend
[20:00:48] <paulej72> crap
[20:01:26] <paulej72> does this have to be a join?
[20:01:34] <NCommander> paulej72, the problem is story_param
[20:01:43] <NCommander> `value` mediumtext COLLATE utf8_unicode_ci NOT NULL,
[20:01:56] <NCommander> Because that's a text and not a varchar, it causes a cascade everytime anything looks it up
[20:02:15] <paulej72> story_text is blob too
[20:02:28] <NCommander> Its not being used in the WHERE clause
[20:03:11] <NCommander> I think
[20:03:15] * NCommander is fiddling with it
[20:04:31] <NCommander> paulej72, mysql> SELECT stories.stoid, title, sid, stories.tid FROM story_text, stories WHERE stories.stoid = story_text.stoid AND in_trash = 'no' AND stories.stoid != '5941' GROUP BY stories.stoid ORDER BY time DESC LIMIT 1\g
[20:04:31] <NCommander> +-------+------------------------------------------------------------------------------------+-----------------+------+
[20:04:31] <NCommander> | stoid | title | sid | tid |
[20:04:31] <NCommander> +-------+------------------------------------------------------------------------------------+-----------------+------+
[20:04:34] <NCommander> | 6407 | UK Shinewald Report Urges Treaty Forcing US Web Firms' Cooperation in Data Sharing | 15/06/02/186207 | 3 |
[20:04:37] <NCommander> +-------+------------------------------------------------------------------------------------+-----------------+------+
[20:04:40] <NCommander> 1 row in set (0.06 sec)
[20:04:47] <NCommander> paulej72, yeah, checking for neverdisplay is what causes the query to hang so badly.
[20:06:02] <NCommander> paulej72, my guess is the simplist solution here is just to change story_param to a varchar
[20:06:31] <paulej72> story_param has the related_sids
[20:06:43] <paulej72> sids are long
[20:07:07] <NCommander> ugh
[20:07:08] <paulej72> 19 chars each
[20:07:15] <NCommander> varchar can go to 255
[20:07:36] <paulej72> ~12 then
[20:08:10] <paulej72> can we query to see if the values are longer than 255?
[20:08:20] * NCommander isn't sure
[20:09:26] <NCommander> +---------------+----------+
[20:09:26] <NCommander> | LENGTH(value) | COUNT(*) |
[20:09:26] <NCommander> +---------------+----------+
[20:09:26] <NCommander> | 1 | 6477 |
[20:09:28] <NCommander> | 2 | 47 |
[20:09:28] <NCommander> | 3 | 660 |
[20:09:30] <NCommander> | 4 | 5569 |
[20:09:31] <NCommander> | 7 | 5838 |
[20:09:33] <NCommander> | 8 | 1 |
[20:09:35] <NCommander> | 10 | 26 |
[20:09:37] <NCommander> | 11 | 21 |
[20:09:39] <NCommander> | 12 | 9 |
[20:09:41] <NCommander> | 13 | 9 |
[20:09:43] <NCommander> | 14 | 12 |
[20:09:45] <NCommander> | 15 | 56 |
[20:09:47] <NCommander> | 16 | 281 |
[20:09:49] <NCommander> | 17 | 3 |
[20:09:53] <NCommander> | 18 | 36 |
[20:09:55] <NCommander> | 19 | 33 |
[20:09:57] <NCommander> | 22 | 2 |
[20:09:59] <NCommander> | 23 | 1 |
[20:10:01] <NCommander> | 25 | 8 |
[20:10:03] <NCommander> | 27 | 1 |
[20:10:05] <NCommander> | 28 | 3 |
[20:10:07] <NCommander> | 29 | 1 |
[20:10:09] <NCommander> | 32 | 10 |
[20:10:11] <NCommander> | 33 | 50 |
[20:10:13] <NCommander> | 34 | 1 |
[20:10:15] <NCommander> | 35 | 2 |
[20:10:17] <NCommander> | 37 | 1 |
[20:10:19] <NCommander> | 39 | 1 |
[20:10:23] <NCommander> | 43 | 1 |
[20:10:25] <NCommander> | 47 | 2 |
[20:10:27] <NCommander> | 48 | 4 |
[20:10:29] <NCommander> | 49 | 3 |
[20:10:31] <NCommander> | 50 | 13 |
[20:10:33] <NCommander> | 56 | 1 |
[20:10:35] <NCommander> | 66 | 4 |
[20:10:37] <NCommander> | 67 | 6 |
[20:10:39] <NCommander> | 69 | 1 |
[20:10:41] <NCommander> | 82 | 1 |
[20:10:43] <NCommander> | 84 | 3 |
[20:10:45] <NCommander> | 86 | 1 |
[20:10:47] <NCommander> | 100 | 1 |
[20:10:49] <NCommander> | 101 | 1 |
[20:10:53] <NCommander> | 385 | 1 |
[20:10:53] <paulej72> 3 stories with >255
[20:10:55] <NCommander> | 421 | 1 |
[20:10:57] <NCommander> | 435 | 1 |
[20:10:59] <NCommander> +---------------+----------+
[20:11:01] <NCommander> paulej72, "fuck"
[20:11:03] <NCommander> neverdisplayed needs a change in venue
[20:11:05] <NCommander> Move it to the stories table, and make it a boolean, so the JOIN doesn't deadlock cluster
[20:11:14] <paulej72> 153, 1742, 67
[20:11:30] <NCommander> probably cases where we put a related link in the box
[20:11:52] <NCommander> Two possibilities
[20:12:19] <paulej72> probably where you added a bunch of links back to SN in one of your monster stories
[20:12:20] <NCommander> We can drop the join on the inital pull, then get them from story_params as a second query
[20:12:34] <NCommander> Or move neverdisplayed
[20:12:58] <NCommander> I'm guessing there are other queries sputtering due to this
[20:13:23] <paulej72> brb
[20:13:33] <NCommander> k
[20:15:59] * NCommander is debating the easiest way to fix this
[20:16:55] <paulej72> there is s shitload of neverdisplay code
[20:17:47] <paulej72> damn one of them is in search code
[20:18:06] <paulej72> on of the bad juju queries
[20:18:06] <NCommander> paulej72, its only a problem when its in a join or where clause
[20:18:40] <NCommander> The problem is O(n), its only when its loading a massive amount of rows do we hit slowdowns
[20:18:49] <paulej72> the code i was worried about is if we move the data to a diffent table
[20:24:08] <paulej72> holy shit batman, they should have never made this a param, it should have been a table col in stories the whole fucking time
[20:26:07] <NCommander> paulej72, I'll bet dollars to donuts that it used to be in early slash, and then someone decided to "clean up the database"
[20:26:50] <paulej72> all for a few bytes
[20:27:29] <NCommander> paulej72, well, querying on text in general is a *bad* idea.
[20:27:41] <NCommander> MySQL really shouldn't even allow it, but it does, and the performance is bad
[20:30:59] <NCommander> paulej72, ok, quick and dirty fix
[20:31:02] <paulej72> NCommander: sub query that pulls up the neverdisplay article
[20:31:17] <NCommander> paulej72, we can query for the EXISTANCE of the neverdisplay key without causing query breakage
[20:31:29] <NCommander> SELECT stories.stoid, sid, stories.tid FROM story_text, stories LEFT JOIN story_param ON (stories.stoid=story_param.stoid AND story_param.name='neverdisplay') WHERE stories.stoid = story_text.stoid AND in_trash = 'no' AND story_param.stoid IS NULL AND stories.stoid != '5941' GROUP BY stories.stoid ORDER BY time DESC LIMIT 1\g
[20:31:29] <NCommander> +-------+-----------------+------+
[20:31:29] <NCommander> | stoid | sid | tid |
[20:31:29] <NCommander> +-------+-----------------+------+
[20:31:31] <NCommander> | 6407 | 15/06/02/186207 | 3 |
[20:31:33] <NCommander> +-------+-----------------+------+
[20:31:35] <NCommander> 1 row in set (0.08 sec)
[20:32:04] <NCommander> I wonder if a subquery could be used ...
[20:32:05] <NCommander> hrm
[20:32:26] <paulej72> NCommander: what about the time clauses
[20:33:10] <NCommander> paulej72, no performance penality for them
[20:33:22] * NCommander realizes he tested the wrong query
[20:33:24] <NCommander> stand by :)
[20:33:46] <paulej72> I just tested one and it ran .10 s
[20:34:05] <NCommander> mysql> SELECT stories.stoid, sid, story_text.title, stories.tid FROM stories LEFT JOIN story_param ON (stories.stoid=story_param.stoid AND story_param.name='neverdisplay'), story_text, story_topics_rendered WHERE stories.stoid = story_text. AND stories.stoid = story_topics_rendered.stoid AND '2015-04-27 13:16:00' > DATE_SUB('2015-06-02 19:31:00', INTERVAL 120 DAY) AND time < '2015
[20:34:05] <NCommander> -04-27 13:16:00' AND time <= '2015-06-02 19:31:00' AND in_trash = 'no' AND story_param.stoid IS NULL AND story_topics_rendered.tid = 1 AND stories.stoid != '5941' GROUP BY stories.stoid ORDER BY time DESC LIMIT 1;
[20:34:05] <NCommander> +-------+------------------+------------------------------------------------------------------------------+------+
[20:34:08] <NCommander> | stoid | sid | title | tid |
[20:34:11] <NCommander> +-------+------------------+------------------------------------------------------------------------------+------+
[20:34:14] <NCommander> | 5940 | 15/04/27/1023237 | Robots Step Into the Backbreaking Agricultural Work that Immigrants Won't Do | 18 |
[20:34:17] <NCommander> +-------+------------------+------------------------------------------------------------------------------+------+
[20:34:20] <NCommander> 1 row in set (0.12 sec)
[20:34:48] <paulej72> NCommander: are we sure that neverdisplay gets removed if neverdisplay is unset
[20:35:01] <NCommander> No
[20:35:12] <NCommander> that's why I'm screwing with a subquery
[20:35:26] <NCommander> but I'm guessing not
[20:35:59] <NCommander> Path of least resistance is probably just making sure neverdisplay gets deleted, and just checking values
[20:36:07] <NCommander> er, for the key
[20:36:10] <NCommander> and ignore the value
[20:36:45] <NCommander> paulej72, or ...
[20:36:45] <NCommander> mysql> select * from story_param where name = 'neverdisplay' and value = 0;
[20:36:45] <NCommander> Empty set (0.09 sec)
[20:36:45] <NCommander> mysql>
[20:37:02] <NCommander> mysql> select count(*) from story_param where name = 'neverdisplay' and value = 1;
[20:37:02] <NCommander> +----------+
[20:37:02] <NCommander> | count(*) |
[20:37:02] <NCommander> +----------+
[20:37:04] <NCommander> | 86 |
[20:37:04] <NCommander> +----------+
[20:37:06] <NCommander> 1 row in set (0.08 sec)
[20:37:07] <NCommander> Appears mysql does that for us \o/
[20:37:09] <NCommander> er
[20:37:11] <NCommander> rehash
[20:37:53] <NCommander> paulej72, I think its safe just to check for the existence of neverdisplay
[20:38:00] <paulej72> yep
[20:39:23] <NCommander> paulej72, actually, quite a bit of code just looks for the key
[20:39:26] <NCommander> sub displaystatusForStories for isntance
[20:39:45] <paulej72> sure
[20:40:03] * NCommander notes this is a hack, but at least site performance will get out of the shitter
[20:40:39] <paulej72> serch uses just neverdisplay in where clause
[20:41:14] <paulej72> Not so sure how big of hack it is but to fix this with just one minor change.
[20:42:04] <NCommander> paulej72, index.pl also does a query for neverdisplay's value
[20:42:10] <NCommander> Probably why the index loads so slowly
[20:42:49] <paulej72> no join on that query is there?
[20:43:17] <paulej72> MySQL.pm 2804
[20:43:31] <NCommander> paulej72, no, but I'd prefer consistent behavior :)
[20:43:36] <NCommander> paulej72, http://pastebin.com - here's my diff thus far
[20:43:36] <aqu4> ^ "3slash@lithium:~/src/rehash-ncommander$ git diff diff --git a/Slash/DB/MySQL/MyS - Pastebin.com"
[20:43:40] <paulej72> yep
[20:45:10] <NCommander> paulej72, well .. that's a drastic improvement even with dev
[20:45:13] <NCommander> Page loads are near instant now
[20:46:07] <paulej72> holy fuxk that is nice
[20:46:48] <paulej72> lets do a pull and get this on prod
[20:46:52] <NCommander> Yeah
[20:48:54] <paulej72> Then we should test if hydrogen is still a dog. but we can't have both in prod just yet as we have rss files being generated by slashd
[20:51:34] <NCommander> paulej72, would you kindy? https://github.com
[20:51:35] <aqu4> ^ "3Allow neverdisplay queries to properly optimize by disable checking t… by NCommander · Pull Request #80 · SoylentNews/rehash · GitHub"
[20:52:26] <paulej72> merged
[20:52:43] <paulej72> Want me to deploy on fluorine
[20:52:57] <paulej72> I ahave it up
[20:52:59] <NCommander> paulej72, well, the proper way to do it is pull on helium and run the deploy script ;)
[20:53:09] <NCommander> which rsyncs all the files and such
[20:53:35] <paulej72> which script for that?
[20:53:44] <NCommander> paulej72, ~slash/bin/deploy-rehash
[20:53:55] <NCommander> paulej72, we may want to bump the rehash version to 15.05.1
[20:54:00] <paulej72> i did not see the rscync part
[20:54:02] <NCommander> Figure this is worth an announcement
[20:54:13] <NCommander> paulej72, its in a subscript
[20:54:29] <NCommander> rsync --progress -a /srv/soylentnews.org $node:/srv
[20:54:30] <NCommander> er, no
[20:54:31] <NCommander> Its there
[20:55:08] <paulej72> rsyncs to itself?
[20:55:26] <NCommander> No, look at $node, it rsyncs from /srv/soylentnews.org to each web frontend
[20:55:33] <NCommander> Its how I deployed fluorine and hydrogen ;)
[20:56:34] <paulej72> from helium i see
[20:56:49] <NCommander> yeah
[20:56:56] <NCommander> searchd/indexer are on helium too
[20:58:20] <paulej72> one problem apache config on helium is broken and plugin and htdocs folders should be wiped/
[20:59:01] <paulej72> stuff got copied from dev that should not have. It is fixed on flourine but not helium
[20:59:12] <NCommander> paulej72, -_-;
[20:59:30] <NCommander> paulej72, do you want to handle deployment and other stuff to try and make shit happy?
[20:59:43] <paulej72> sure
[21:00:28] <NCommander> Let me go when it goes live
[21:00:35] * NCommander is expecting a *drastic improvement to shit*
[21:04:46] <NCommander> O_o;, we already transferred 5 GiB of data this month
[21:05:19] <paulej72> copies of datat to oxygen
[21:07:47] <paulej72> NCommander:
[21:07:47] <paulej72> chown: changing ownership of â/srv/soylentnews.org/rehash/themes/soylentnewsâ: Operation not permitted
[21:10:11] <paulej72> make: *** [install] Error 1
[21:11:03] <NCommander> ugh
[21:11:15] <NCommander> paulej72, I'm lookingat the load balancer alone *not* general output
[21:12:45] <paulej72> on helium,
[21:13:18] <paulej72> NCommander: did a deploy right on fluorine and we are up and running
[21:16:03] <paulej72> NCommander: should we not be using prod_access as the group for rehash install?
[21:18:23] <NCommander> paulej72, oh shit. I forgot that existed >.<;
[21:18:27] <NCommander> (prod_access)
[21:18:41] <NCommander> paulej72, holy shit, production is flying
[21:18:47] <paulej72> yep
[21:19:11] <NCommander> I actually think its faster than the old system
[21:22:46] <paulej72> that bad query was bad before
[21:26:52] <paulej72> NCommander: we have anotyher speed hit. https://soylentnews.org is slow
[21:26:52] <aqu4> ^ "3 - SoylentNews User "
[21:27:41] <paulej72> SET timestamp=1433280433;
[21:27:41] <paulej72> SELECT hits, stories.commentcount AS commentcount,
[21:27:41] <paulej72> stories.stoid, stories.sid,
[21:27:41] <paulej72> story_text.title, stories.uid, stories.tid,
[21:27:41] <paulej72> time, stories.in_trash, primaryskid
[21:27:43] <paulej72> FROM story_text, stories LEFT JOIN story_topics_rendered AS str ON str.stoid = stories.stoid WHERE stories.stoid = story_text.stoid AND (str.tid = 24 OR stories.primaryskid = 2) GROUP BY stoid ORDER BY time DESC LIMIT 0, 40;
[21:33:17] <NCommander> paulej72, true, but slowdowns with admin.pl can be lived with ;)
[21:33:23] <NCommander> paulej72, def should be fixed though
[21:33:31] <NCommander> I did a quick announcement that site performance is back to normal
[21:33:51] <paulej72> # Time: 150602 21:27:13
[21:33:51] <paulej72> # User@Host: prod_writer[prod_writer] @ localhost [] Id: 52584
[21:33:51] <paulej72> # Query_time: 3.339698 Lock_time: 0.000132 Rows_sent: 4 Rows_examined: 38128
[21:35:44] <NCommander> paulej72, we should probably wait a full 24 hours to let the slow log build up, run it through mysqldumpslow, and then fix what we find
[21:35:47] <NCommander> THe critical one got fixed
[21:36:01] <paulej72> sure
[21:36:58] <paulej72> looks like hydrogen may be up to speed now, but i have some bugs that need fixed boefore I put it into prod
[21:38:59] <NCommander> paulej72, I thought we were going to nuke hydrogen from low earth orbit
[21:39:32] <paulej72> that is when I thougt the speed problems wer due to some voodoo on it.
[21:39:51] <paulej72> Testing now seems it may be ok
[21:40:24] <paulej72> it might have only been slow due to db issues
[22:15:29] <paulej72> NCommander: how do i restart mysql on hydrogen
[22:21:47] <NCommander> paulej72, sudo -u mysql /opt/mysql/support-files/mysql.server start
[22:24:35] <paulej72> strange issue on hydrogen : http://198.58.127.22
[22:24:36] <aqu4> ^ "3SoylentNews: SoylentNews is people"
[22:25:15] <paulej72> it shows the section_banner at the top but it should not. fluorine does not.
[22:26:10] <paulej72> it is empty, but it is almost like something is broken on the install
[23:21:47] -!- audioguyalt [audioguyalt!~audioguy@50.38.mt.uk] has joined #dev
[23:25:17] -!- audioguy has quit [Ping timeout: 264 seconds]
[23:28:22] -!- audioguyalt has quit [Quit: Leaving]