Further reading..
https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/webfont-optimization
Looks like the browsers (some of them at least) only download the parts of the font needed to show the page. There's an explanation at the end of the page for this.
I'm looking into getting a subset of the font loaded earlier so the visible content can be output to the browser sooner.