To specify the maximum allowed file size, navigate to Settings → Global Settings → System. Change "Max Upload File Size" to your liking. The value should be specified in bytes. For example, to limit the maximum allowed attachment size to 50Mb, you should specify
50000000.See this article for more information on that.
Note that if there's a reverse proxy installed in front of YouTrack it should also be configured to let big files through.