Fully starting YouTrack after server restart
I'm running YouTrack 2017.1 on a Centos 7.3 server. The YouTrack server is configured to start automatically on server restarts using the method recommended in the documentation under the "YouTrack JAR as a Service on Linux" method.
This mostly works, but it has a significant limitation in that it only partially starts the server, so the first few clients just get server timeouts when they try to connect.
When I boot the server and let things settle down, the youtrack process shows about 1min:20sec of CPU time (from top):
1854 youtrack 20 0 3965488 499008 13888 S 0.0 25.3 1:20.22 java
When you try to connect to the server, the first 2-3 connections time out, and eventually the server stabilizes showing about 3 minutes of CPU time:
1854 youtrack 20 0 4105056 715396 13072 S 0.0 36.2 3:13.35 java
So for the first ~2 minutes after the first HTTP connection after rebooting a server, you can't actually access it, which causes a number of our REST API calls into the server to fail. Now every time we reboot a server with YouTrack we need to ask the admin to refresh the YouTrack page a few times to get it to fully start up, but this seems like a silly thing to need to do .. can I change something in the config so it will start up properly on it's own?
Thanks,
Geoff
Please sign in to leave a comment.
Hi Geoffrey, I'm sorry for the delay.
First, let me clear thing out a bit.
There are two "phases" of application startup: 1) before it opens the http port; 2) everything after that.
Some details follow:
1) Under normal conditions, ~20s (if given enough CPUs) would pass from the moment the process was started to the point when application port starts accepting connections (though the actual cpu time spent is indeed close to 1m).
2) According to current design (and for historical reasons), youtrack application continues to boot once it has opened the port AND any http request to this port was made (said request would hang until the app starts), that includes REST calls.
So, basically, if there are any requests perdiodically coming in to the application port (e.g. background requests from a browser tab with youtrack, or REST API calls), it will eventualy start to boot, and in this case there is no need to manually refresh YouTrack page, since the first request that has passed through would trigger the startup anyway. The fact that application is starting could be verified by lines containing "[init servlet ]" appearing in youtrack.log (actually, youtrack.log is usually not even written into until the application enters this "phase 2").
Given all that, do I understand correctly that your problem is that the so-called "phase 2" of startup takes too long (since you mentioned requests timing out)? Or is it still the fact that application won't continue booting until a HTTP request comes in?
1) If the former, unfortunately, I don't think that we could greatly reduce these timings (since the numbers I see are very close to the results I observe on my machine, though the actual startup time is around 60 seconds for me).
Currently, I could only say that there are some 3 threads active duting init on average, so if you have at least 3 (virtual) processors on your server, I doubt the startup time of your instance could be greatly improved.
By the way, I won't recommend running YT on a machine with 1 CPU in any case.
2) If the latter, there is not much that could be done besides configuring your REST integration to handle timeouts gracefully (if this integration is your own custom solution).
Though I could file a request in our tracker to support the scenario of "fully automatic" startup, I don't think it would be implemented very soon.
In any case, just to be sure that everything is in order, I'd like to see the application logs from your instance startup, as well as the startup parameters (which should be located in wrapper.conf), if you don't mind.
Also, could you be so kind as to send the statistics of your instance (in the form of a Screenshot of /admin/statistics page)?
Hi,
I'm definitely in case 2 here, which takes about 2 minutes on our machine. I don't know I can make the client REST API timeout that long, even though it is under my control.
Right now, I'm running this on a Microsoft Azure Linux VM. It was previously on a Basic A1 instance, now it's on a Basic A1_v2 instance - both of which are single-CPU VMs. You can get a multi-processor VM in Azure, but it's basically twice the price .. so I don't see us going there. As other people have commented in the forums, memory consumption of YouTrack is a big challenge on these servers as the cheap VM's don't come with much RAM.
I mainly wanted to confirm this is the expected behavior - which it seems to be based on your explanation. Knowing that, I think we'll play with your startup script a bit and see if we can trigger a wget locally in the startup script to kickoff phase 2 once phase 1 has been given enough time to finish. Not the most elegant solution, but seems like the simplest thing in the short term. If I get that working (doesn't seem like it should be hard), I'll post what we did back here.
Hi,
It's pretty ghetto, but it does seem to work reliably on my server. I updated the /etc/init.d/youtrack startup script in the start() block to wait for the server to start and then do wget to kick off phase 2.
start() {
echo "Starting $APP_LONG_NAME..."
getpid
if [ "X$pid" = "X" ]
then
# The string passed to eval must handles spaces in paths correctly.
COMMAND_LINE="$CMDNICE \"$WRAPPER_CMD\" \"$WRAPPER_CONF\" wrapper.syslog.ident=$APP_NAME wrapper.pidfile=\"$PIDFILE\" wrapper.daemonize=TRUE $ANCHORPROP $IGNOREPROP $LOCKPROP"
echo $COMMAND_LINE
eval $COMMAND_LINE
COMMAND_LINE="/bin/sleep 240"
echo $COMMAND_LINE
eval $COMMAND_LINE
COMMAND_LINE="/bin/wget -r -q -nd -b --timeout=300 https://<server url/"
echo $COMMAND_LINE
eval $COMMAND_LINE
else
echo "$APP_LONG_NAME is already running."
exit 1
fi
}