Server response time and page speed are two different metrics, which both have a large impact on user experience and search rankings.
Server response time also known as Time to First Byte (TTFB) is the total amount of time spent to receive the first byte of the response once it has been requested.
It is the sum of "Redirect duration" + "Connection duration" + "Backend duration". This metric is one of the key indicators of web performance. Ideally under 200ms (milliseconds) server response time is the standard.
Having more content on a page increases the page load time which has a negative impact on user experience and search rankings but provides the user with a more informative experience so a balance is needed to be struck between the volume of informative content and page load speed for user experience and search rankings to increase.
A way around lowering page load times is by spreading out content on a site so that not one singular page contains all of the site's best/most popular content. However, lowering server response times is dependent on the hardware used to host the site and therefore cannot be lowered so easily.
Hosting your site on high-end, servers can produce server response times of less that 10ms (milliseconds) and bolster user experience and search rankings massively, a service which can be found here at Bison Grid.