What opinion do Mission residents have about bicycling lanes?
Like many cities in Canada and the United States, Mission asks residents for their opinion on the quality of services they receive from the municipality. Mission's office of Civic Engagement and Corporate Initiatives makes available a "citizen satisfaction survey" for residents to complete prior to budget deliberations, in the summer. This link to the 2017 survey results provides an example of the range of questions, and responses, that are obtained. The bulk of the survey asks respondents to rate a list of 25 "service areas", including "cycle routes", with regard to their 1 )importance, 2) perceived quality and 3) opinion on whether the level of service should be increased, maintained or reduced. Mission's satisfaction survey therefore provides an opportunity to gauge support of bicycle route infrastructure.
An important caveat, however, is that the survey is voluntary and therefore not necessarily representative of the population as a whole. Statistically valid surveys are based on population sampling such that every resident or household has an equal probability of being included in the survey. Voluntary surveys are unscientific simply because they represent the opinion of only those who decide to opt in, not the population as a whole. How different is the opinion expressed in a voluntary survey from a scientific survey? This is the question I gave to my quantitative methods class at UFV in the fall semester, 2019. I've taught a version of this course in the Geography Department since 1995 and am always on the lookout for term assignments that have local application. The assignment was prepared in collaboration with Mission's Director of Civic Engagement at the time, Michael Boronowski. In summer 2019, using municipal address data, a random sample of 2000 households was selected and mailed an invitation to complete the satisfaction survey. At the same time, residents were able to voluntarily complete the same survey through the municipality's website.
The District received approximately 250 completed surveys. The response rate was disappointing (12.5%) and we didn't have the resources to follow up with the sample to encourage more completions. However it is important to keep in mind that the absolute number of responses does not have to be large in order to be representative of the wider population. Opinion polls for Canada as a whole are typically based on fewer than 2,000 responses. Also, the 250 respondents in the Mission survey was similar to the number of volunteers who answered online. If nothing else, this provides a bit of a balance in the two data files.
To help with analysis, the class created a scorecard for each of the three categories of assessment. These are shown in the figures, below. The scorecard shows the 'average' score for each service that was obtained from each of the two groups of respondents, 1) voluntary and 2) sampled. With this type of data, known as ordinal, the calculation of an average value is determine by the value of the median. To calculate the median, numbers are assigned to the possible answers (eg, very important=1, important=2, etc)and all of the responses are arranged in descending order. The median is the answer which is located in the middle of this list. A related measure is the mode; this is the answer chosen by the largest number of respondents. A visualization of these measures is shown in Figure 2. This figure shows how the two groups rated the importance of cycle routes. Surveyed respondents tended to rate cycling routes as an "important" service and over 40% (the modal category) chose "very important". In contrast, the graph for voluntary respondents was loaded up on the other side of the ledger: "not important" and "not at all important". However, the average was on the fence: "neither important or unimportant", which was also the modal category.
This difference in opinion on the importance of cycling routes in Mission is notable partly because it was otherwise rare for the two groups to differ on the rating of anything. In fact, opinions were identical for 87% of the ratings, as reflected in the large number of check marks in the three scorecards. Besides cycling lanes, there were also differences in opinion on three environmental services. However, here it was with the level of importance: "important" vs. "very important". A few differences are also visible in scorecards 2 and 3. With cycling lanes though, both groups tended to select "neutral" on quality and to "maintain" their service level.
To summarize:
voluntary surveys are, by their nature, not statistically representative of the wider population from which they are taken because they are not random samples.
in a class assignment, I had my second-year Geography students compare the results of an opinion survey from a random sample of Mission residents with a volunteer sample.
the results from the two groups were very similar: there was an 87% overlap.
opinion on bicycle routes was one of the few topics that produced a difference in opinion: the sampled group rated them to be important whereas the volunteer group were undecided: they rated them to be neither important nor unimportant.
Figure 1: Scorecard 1 - Importance of service
Figure 2: Distribution of opinion, "importance of cycling routes", surveyed vs. voluntary.
Figure 3: Scorecard 2 - Quality of service
Figure 4: Scorecard 3 - Increase, decrease or maintain service level