A VPN could provide for isolating any infrastructure not intended for public access.
I'm not aware of the "matrix" so am unable to comment on suitability - Is that open source? IMHO XMPP is a much more viable solution. It's open source, throughly tested, pretty much bullet proof in terms of reliability, resilient against assault and able to operate in a distributed fashion in as much as me being connected to my XMPP server doesn't isolate me from users on other XMPP servers. Settings dependant ofc. It supports textual IM, vid, voice, files - anything you can wedge into an XML stanza. Native SSL/TLS can be further wrapped in GPG/PGP and additional layers of OTR to ensure privacy and assured authentication of conversation recipient. It supports 1:1, 1:many and many:many comms.
There's not a lot it can't do really.