Here is the report from our host on what brought us down today. How would you guys recommend getting this fixed up. Thanks!
Kinsta:
Re: Performance Issues
We've checked your site's performance issues once more. Our system engineers have looked into the public infrastructure where your site is hosted, they checked your site's server and your site's server container. As mentioned previously, we do see a larger spike in CPU usage at around 3pm UTC / 7am your time. Our team looked into possible reasons here and as we mentioned earlier in the conversation the main thing that stood out is a very high amount of traffic going to the following wpdiscuz wp-json URL:
If we check the 30 days but exclude today, these URLs are not all on top.
We did check the traffic specifically, I checked if any bot-like activity is visible there. We did however not find any suspicious traffic so far. This does not mean that this is not necessarily the case as sophisticated bots can definitely camouflage very well but from the way the URLs are looking, there's just no concrete evidence in that direction. If you or your team do however see any indicators for that, we will of course check further.
Another important point about these wpdiscuz URLs is that they are all bypassed from cache. Three things about this are important to understand:
Bypassed (from cache) pages will load via PHP and MySQL directly. This means the server resources are impacted as the server-side cache is not serving that page.
Due to the fact that you have WP-Cron running when any site is opened via PHP/MySQL with these requests, WP-Cron is also being run in the background. We do highly recommend to disable WP-Cron running this way and only let it run from the server-side cronjob. We do have an article about this here which I recommend you and your team check out.
Due to PHP/MySQL being used in the initial request of said URL and in the requests made to WP-Cron, this results in very high PHP/MySQL usage which impacts the resources on the rest of your site which is causing the slowness you have experiences (specifically in WP-Admin where every page has to be uncached for WP-Admin to work properly.)
Our suggestion here is to cache requests to this URL there if that is possible for you. We can set this up if you want to. We would alsorecommend that you contact the plugin developer of wpdiscuz and let them know about both the traffic you're seeing as well as the frequency in which the URL is opened. They might be aware of specific workarounds and possibilities to optimize how the plugin and specifically the wp-json aspect of the site runs to combat the issue.
It seems you've enabled live update and your website was under DDoS attack, so you've got tons of live update request. Please navigate to Dashboard > wpDiscuz > Settings > Live Commenting and Notifications Tab, and configure following settings:
1. Disable the live update for non-authorized visitors and request by disabling the "Enable Live Update for Guests" option.
2. You can keep enabled the "Comment Bubble Live Update" option but disable "Live Update" option.
3. Increase the period of new comment checking requests, set 1 min or higher the "Update Comment List Every" option.
At no point was there any evidence of a DDoS and Kinsta never indicated that either, so please let's not bring any unsubstantiated variables into the discussion.
Currently, we have disabled the WPDiscuz plugin and it is still making requests.
1. This was already disabled
2/3/4: None of this had any impact on the site.
As you can see in the conversation below, WPDiscuz is still making calls while we have disabled the plugin. Please provide instructions to stop the adverse impact on our website. Thanks.
***KINSTA TRANSCRIPT
We disabled those plugins in a past trouble shoot and those calls kept coming according to a past rep
Yeah, I'm seeing them right now.
There's a lot of them. Since it's not being used, can I 403 those requests? I think that will immediately relieve some pressure from the site since WP is still trying to process those, even if it's only to say 'Nothing is here'
so we intend to use that plugin, how will that impact our use going forward? Is there anything I can report to them that the plugin shouldn't be doing?
Hm.. we shouldn't 403 it then if you want to use it. Right now WordPress is returning a 403 error, but it's having to use PHP to render those responses so it's using up a PHP Worker on each request.
It's 403 because the plugin is disabled, right now.
I don't think it could be anything cached, as you'd need to be logged in to view that page.
It looks like the last edit you may have done was slow due to a lot of things happening in the background externally.
migtrate2wpforo is a related plugin ... same company ... but the forums software basically
wpdiscuz is a commenting plugin
Yeah, I don't think it inherently has any problems, it was just processing a lot of data on that request. It's constructor seemed to take a while which runs when the class is initialized.
Right now on average I'm seeing between 0.6 and 0.7 seconds for requests made to the site.
There's still a flood of them hitting wp-json/wpdiscz/v1/update which is very odd.
All of those wp-json requests are certainly going to tie up php workers though.
is there anything you can give me that I can provide to WPdiscuz that really highlights the issue? Because if we're using them we need to figure out the caching issue
It seems to be coming from Javascript, so there's a chance that's cached somewhere 🤔
Minified JS could do it. Any plugins installed to minify Javascript?
I think it would be good to try to decrease the amount of requests the site is making to wp-json/wpdiscuz/v1/update if possible, to be honest, they shouldn't be showing up right now anyway with the plugin disabled, but you're wanting to use that plugin in the future so I would definitely reach out to the plugin developers about it to see how its usage can be less invasive/resource intensive.
yeah i thought about that when thinking about resources in general, but dod you think it's still going while it's disabled?
I think somewhere some javascript probably got minimized to include that call which is why we're still seeing it even though the plugin is disabled.
I don't see any minified javascript files in the wp-content directory though, so it's likely in the database.
hmmmm
It wouldn't be our cache, since once you log in, you're no longer served any Nginx cache.
interesting
We don't want to cache those calls because the plugin won't work
Right, it would take 1 hour to update if we cached them.
As you can see though, it's the top requested URI on the site.
I wonder if there's any way to kill those calls without permanently impact
or otherwise some way to make those calls less resource intensive
I've got an idea - You're using Kinsta's CDN which would serve up javascript and other cached assets like images. It's very possible that's where the caching is happening at.
Can we clear the Kinsta CDN cache and let it repopulate?
for sure
Great, let me do that and I'll keep monitoring the incoming requests to see if it decreases.
🙌
Do you have any objections if I make a Kinsta user to view the pages it's saying the request is coming from so I can inspect the source code I'm given?
None at all. Thank you for all this great service!
A lot of what I'm looking into really is getting very much outside the scope of what we support. I know I was pretty clear about that earlier. I do want you to have a good experience though and I understand how this is affecting you, so I'll do what I can but it is indeed outside of what we normally would support.
On my user login on some of those pages that are referred to as the 'referer' page, I'm not seeing anything relating to wpdiscuz in any of the requests my browser is making or in the source code. 🤔
It could be stored in users browsers or perhaps they're just sitting on that page and haven't refreshed so it's trying the call over and over since it's not getting the response it wants.
right ... which would be a plugin issue
I think we've covered a lot of good ground here
I'd like to hear what the plugin maker has to say about this
The js files are being cached on your website, so you should properly delete all kind of caches and purge CDN if you have, then press Ctrl+F5 (twice) on front-end to make sure the browser caches are deleted as well.
In case you want to say thank you! 🙂 We'd really appreciate if you leave a good review on the plugin page. This is the best way to say thank you to this project and the support team.
Our site has been brought down for a week and our host has pointed to your plugin as the culprit and you can't even respond? Not a great look for your company.
Please don't duplicate the questions under each topic. If you have some issues, please just open your own support topic and wait for the answer.
In case you want to say thank you! 🙂 We'd really appreciate if you leave a good review on the plugin page. This is the best way to say thank you to this project and the support team.
Sorry about that but as you can see, I'm replying to the people that posted. I'm not creating any new topic. I'm just asking them if they managed to resolve the issue. So, far, it's not affecting me but if I have to use WPDiscuz on a high traffic site, such an issue should be of worry.
Was this ever resolved or is the "solution" to simply disable the live update function on sites that have cache?
I just enabled the live update and while my server didn't crash, the 404 errors were cramming the log files.
Seems like gVectors has a bug they need to address here and it would be nice to see it addressed as such rather than trying to blame the hosting company. Or at least a tacit acknowledgement that the live update feature is incompatible with server cache.
