Performance issues

Hi,

We have been using standalone Youtrack installation for over a year, the performance slowly degraded while number of issues increased, however recently it is getting unbearable slow - it can take even half a minute to get a list of issues and several seconds to load the single issue.

Our Youtrack installation (version 5.1) is running on a dedicated server with x64 Ubuntu 12.04 installed, 2x8 core Xeons, 4GB of RAM, 2 SAS drives in RAID1. We use Tanuki Java Service Wrapper to launch the software. The Java heap memory is currently set to 3GB, but increasing it to 3.5GB does not seem to change anything.

I Have not found anything hints in log, apart from the following message being written several times a day "jetbrains.charisma.main.ProcessingTookToLongException: Processing took too long :(". The slowdown does not seem to correlate with number of users - even with few of them active we still see a performance degradation.

I put Apache2 caching proxy in front of Youtrack to cache at least static files, but it did not help either.

Is there anything else which can be done, or have we outgrown our hardware?

Regards,
Marcin

Telemetry details:
Available processors     8
Available memory     3125.5 MB
Allocated memory     3125.5 MB
Used memory     2089.6 MB
Uptime     1 hour, 46 minutes, 57 seconds and 299 milliseconds (started at Thursday, July 3, 2014 7:55:14 AM UTC)
Online users     Users: 15 (max is 16, 44 minutes ago (08:57))
Sessions: 21
Windows: 54
Database size (without blobs)     6.7 GB
Text index size     34.4 MB
Database background threads     2
Pending asynchronous jobs     0
Number of cached results in the database queries cache     3389
Database queries cache hit rate     84.40%
Blob strings cache hit rate     81.30%
Total transactions     1109
Transactions per second     0.17
Requests per second     1.92
Notification Analyzer Queue Size     0
Notification Sender Queue Size     0
User Action Job Processor Queue Size     0
There is no job being executed
4 comments
Comment actions Permalink
Hi Marcin,

Thank you for reporting this. What's the total number of issues?
Please, try to start YouTrack with parameter:
-Dexodus.gc.utilization.fromScratch=true
prettyPrint();
After that, please check if the database size decreases or not.
Normally, with the database size increasing RAM is required. Could you also, please send us log to youtrack-feedback@jetbrains.com ?

Thank you.
0
Comment actions Permalink
Hi Andrey,

Many thanks for your reply.

I did start Youtrack with the parameter you suggested, what is worth mentioning is that it took ~40 minutes since launch to the first request served. I also noticed unusually high disk activity during the start. Performance-wise we have not observed a noticeable improvement.

The database size has not changed significantly either, please find current telemetry details below. In regards to logs, I have just sent them via email to indicated email address.

The total number of issues is around 12 thousand.

Best regards,
Marcin

Available processors     8
Available memory     3125.5 MB
Allocated memory     3125.5 MB
Used memory     2094.5 MB
Uptime     68 hours, 2 minutes, 58 seconds and 967 milliseconds (started at Friday, July 4, 2014 12:13:27 PM UTC)
Online users     Users: 13 (max is 14, 2 days and 20 hours ago (04 Jul 2014 12:13))
Sessions: 16
Windows: 38
Database size (without blobs)     6.6 GB
Text index size     18.3 MB
Database background threads     2
Pending asynchronous jobs     0
Number of cached results in the database queries cache     3605
Database queries cache hit rate     97.16%
Blob strings cache hit rate     99.32%
Total transactions     9316
Transactions per second     0.04
Requests per second     1.23
Notification Analyzer Queue Size     0
Notification Sender Queue Size     0
User Action Job Processor Queue Size     0
There is no job being executed
0
Comment actions Permalink
Hello Marcin,

Thank you for details, we've been able to investigate the issue.
Seems, like your database increases unexpectedly. Normally, it should be of a smaller size.
I assume two solutions- we've actually implemented valuable database performance improvements in 5.2.2, so the problem should be fixed after upgrade; either you may double your Java Heap size.

Thank you.
0
Comment actions Permalink
Hi Andrey,

Thanks for looking into this.

I replied via *-feedback@ email address, but it seems like my email did not reach you.

I do not think increasing Java Heap size is the way to go forward - the database size (by which I understand the total size of all *.xd files located in the teamsysdata directory - obviously excluding the entire blobs subdirectory) has increased by nearly a 1GB since July 1st. Yet, there were only 702 issues modified within that period.

Is there anything we could use to "shrink" or otherwise "optimize" Youtrack database? I know it is supposed to be self-managing, but it looks like something has failed and it just grows infinitely.

Regards,
Marcin
0

Please sign in to leave a comment.