How an attacker can give an access to the file system when only 25 and 587 ports are available from outside? How is it possible via the SMTP protocol? Does it have vulnerabilities?
A protocol usually doesn't, but certain implementations may. Specifically, the "Sendmail" SMTP suite1 (which used to be the major Unix SMTP implementation for a long time) had a few security holes, some of them very famous examples at that.
Postfix was deliberately created to become a secure alternative to Sendmail, and much of its design – the usage of chroots, the multi-process architecture with strong separation between tasks – came directly as a response to the security issues that Sendmail used to have. (Sendmail consisted of one process that did everything, and it used to handle SMTP while running as root!)
1 Not to be confused with the 'sendmail' program in your /usr/lib, which indeed used to be Sendmail back then, but the one on your system is still Postfix and merely follows the interface of the original Sendmail.
Why Postfix is highly recommended to install in a chroot, but Dovecot that publishes POP3/IMAP ports and stores emails in the file system is not?
Dovecot does use chroot. Its architecture (as well as Postfix's, really) allows it to be done in a more fine-grained manner, so that e.g. the highly-exposed "login" processes can do this automatically, so it becomes a little less important to do the same service-wide. But POP/IMAP also makes it somewhat more complex than SMTP, as it needs to access mailboxes stored in user home directories (which was the traditional layout), so a single chroot wouldn't work for that kind of setup.