To specify the maximum allowed file size, navigate to Settings > Global Settings > System. Change "Max Upload File Size" to your liking. The value should specified in bytes. For example, to limit the maximun allowed attachment size to 50Mb, you should specify
If there's a reverse proxy installed in front of YouTrack it should also be configured to let big files through.