YouTrack uses 300% CPU and hangs up every other day

Hi Stephen,

Could you please provide me with your Administration> More> Statistics page?

Thank you.

Sure:

edit: I hope you can see the attached screenshot.
http://i.imgur.com/26Cic02.png?1

I think I found the reason for the problems:

I use haproxy in front of youtrack and I had level 4 health checks enabled, which means that the server gets polled every 2s, but that poll doesn't initiate a full http request. I disabled health checks and it seems that the CPU load is normal now.

Well, disabling the health checks improved the situation, but Youtrack is still causing a high load:

Instead of 300% cpu load, Youtrack causes only ~140%. And used 18h cpu time in the last 22h.

edit: I can see a few exceptions like this one:

Jan 14 12:28:12 QCGK001 youtrack[19302]: 12:28:12,276 WARN  [QueryOperations] [tp1264387597-34] [phil@::ffff:x.x.x.x] Brute force calculation of roughCount! Jan 14 12:28:12 QCGK001 youtrack[19302]: java.lang.Exception Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.teamsys.dnq.runtime.queries.QueryOperations.roughCount(QueryOperations.java:142) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.charisma.customfields.complex.user.UsersBundleImpl.isLazyLoadElementsForDraft(UsersBundleImpl.java:257) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.charisma.customfields.complex.user.UserFieldWrapper.lazyLoadValues(UserFieldWrapper.java:25) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.charisma.customfields.complex.user.SingleUserSelectorEditor_HtmlTemplateComponent.renderTemplate(SingleUserSelectorEditor_HtmlTemplateComponent.java:173) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateComponent.renderTemplate(TemplateComponent.java:474) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateComponent.buildTemplateComponent(TemplateComponent.java:804) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.charisma.smartui.fullScreenIssue.VerticalProperties_HtmlTemplateComponent.renderTemplate(VerticalProperties_HtmlTemplateComponent.java:341) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateComponent.renderTemplate(TemplateComponent.java:474) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateComponent.render(TemplateComponent.java:391) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateComponent.getRefreshContent(TemplateComponent.java:418) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.component.HtmlTemplate$1.generateScript(HtmlTemplate.java:33) Jan 14 12:28:12 QCGK001 youtrack[19302]: at webr.framework.runtime.response.CommandResponseAction.generateScript(CommandResponseAction.java:17) Jan 14 12:28:12 QCGK001 youtrack[19302]: at webr.framework.runtime.response.JsCommandResponse.doGenerateScript(JsCommandResponse.java:64) Jan 14 12:28:12 QCGK001 youtrack[19302]: at webr.framework.runtime.response.JsCommandResponse.generateContent(JsCommandResponse.java:43) Jan 14 12:28:12 QCGK001 youtrack[19302]: at webr.framework.runtime.response.JsCommandResponse.prebuildContent(JsCommandResponse.java:54) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateComponent.prebuildCompositeCommandsContent(TemplateComponent.java:559) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateComponent.prebuildCompositeCommandsContent(TemplateComponent.java:539) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateComponent.prebuildCompositeCommandsContent(TemplateComponent.java:539) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateComponent.prebuildCompositeCommandsContent(TemplateComponent.java:539) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateComponent.prebuildCompositeCommandsContent(TemplateComponent.java:539) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateActionController.processAjaxCommand(TemplateActionController.java:192) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateActionController.handleEventImpl(TemplateActionController.java:166) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.templateComponent.TemplateActionController.handleEvent(TemplateActionController.java:126) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.requestProcessor.EventRequestProcessor.processRequest(EventRequestProcessor.java:75) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.servlet.MainServlet.processRequest(MainServlet.java:259) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.servlet.MainServlet.doGet(MainServlet.java:145) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.servlet.MainServlet.doPost(MainServlet.java:223) Jan 14 12:28:12 QCGK001 youtrack[19302]: at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) Jan 14 12:28:12 QCGK001 youtrack[19302]: at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:648) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1336) Jan 14 12:28:12 QCGK001 youtrack[19302]: at jetbrains.mps.webr.runtime.filter.QueryParameterFilter.doFilter(QueryParameterFilter.java:25) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.continuation.ContinuationFilter.doFilter(ContinuationFilter.java:137) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:242) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.Server.handle(Server.java:365) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:856) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) Jan 14 12:28:12 QCGK001 youtrack[19302]: at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) Jan 14 12:28:12 QCGK001 youtrack[19302]: at java.lang.Thread.run(Thread.java:745)