It is possible to access your internal server using the router’s public IP address/forwarded port, but your router must support loopback network address translation (a.k.a. hairpin NAT) and be configured accordingly.
Without loopback translation in effect, IP packets destined for your router’s public IP address will arrive on its private network interface. Here the packets are simply dropped, usually because the router requires packets destined for its public IP address to arrive on its physical WAN interface (there are other possible ways this traffic may be handled, including NAT occurring, but the end result is always that the connection cannot be completed).
However, with loopback translation configured, when the packets destined for the public address arrive on the private interface, their destination address will be rewritten to be that of the internal server. Then the normal port translation will be applied, further resulting in the destination port being rewritten as well.
Twisty already answered your question as it stands, but as a pointer to what you can do about it; in a similar situation for my LAN I configured the on-lan DNS server (built into the router, or a windows SBS machine I can’t remember which, not authoritative for my domain name; only answers requests from LAN clients because they’re configured to try it first – indeed some LAN machines even just have a hosts file entry alluding to the same) to report downloadsite.mydomain.com as 192.168.1.10, but the actual external authoritative DNS reports downloadsite.mydomain.com as 126.96.36.199
This way someone on the lan can form and test the link downloadsite.mydomain.com/somefile.txt and give it to an external third party, confident that it’ll work out