Author:
Blog About:
Just wanted to give everyone (that's not on Discord) a heads up as to what has gone on the last couple days.
About 2 days ago(?) I was told by others that the site was down. I went to investigate and found the "Front End" servers each with a "load" of 200+
I killed varnish, to let the front ends slow down, I did some maintenance, and then I edited the virtual servers to upgrade them from 4 to 8 cores of processing power.
Once rebooted everything seemed to be fine, and even a bit snappier so I was happy with this result and let it fly.
This morning when I went to do some work, the servers were over-loaded again. I started working on bringing up a 4th front end server to take load off the current 3, but I ran into issues with that and had to do software updates and more maintenance to try and fix things.
And that's where the most recent problem was introduced.
When I processed drupal Module updates, I didn't notice that one of them wasn't working properly and causing instability for certain areas of the site.
So after tracking this issue down, and restoring an older version of the module I was able to get the site back up. I also did even more maintenance on a few things and while the 4th virtual front end isn't fully configured, I did get it spun up and my basic configure scripts run. I just need to finish the micro configs that are unique to each front end, as well as get it into the varnish round-robin.
So this has been my dev blog to let y'all know that we do know stuff has been going on, and that we are definitely working on it.
-Piper, Cat, Erin, Jamie, Amy, NinjaBaby and the BigCloset Band :)
Comments
Log-ins
Would that be why I keep having to log-in every time I change pages? Go to home page, log-in, click on story or comment, after reading click on home page, log-in again, repeat.
"Life is not measured by the breaths you take, but by the moments that take your breath away.”
George Carlin
Lingering Errors...
Sort of... That was another lingering error that I think I was finally able to quash.
-HuGs--Piper
No worries...
I figured that the discrepancies were due to something like that. I've experienced much more hurtful problems. Anyone who complains is a Piss Ant.
Gwen
Really?
You, of course, have NEVER complained. :O
"Life is not measured by the breaths you take, but by the moments that take your breath away.”
George Carlin
I just thought...
...that it was a BC version of the movie "Groundhog Day" and went with it.
Thank you
Once again a heartfelt thankyou to Piper and the team for keeping our home from home up and running tin of Kitty foood en route PP
Eighteen
I understood all of eighteen nouns and verbs in your blog AND that we all owe you a big THANK YOU for all you do.
Jill
Angela Rasch (Jill M I)
Not only does she do all of
Not only does she do all of that, she helps me virtually teach and parent our child, puts up with the fact that I babble at her about psychology or random facebook groups, deals with my "I am never going to graduate! What was I thinking" mini break downs and puts up w/ my family. I don't thank her enough!
Thanks Piper.
You're doing a bloody good job.
xxx Beverly.