Why students find Stack Overflow useful, and when they don't
In one of my junior optional courses, I conducted a simple survey in the form of homework to ask my classmates’ experience about Stack Overflow. They are all my peers at the Software Institute, who have extensive experience of developing large and new software projects (therefore use Stack Overflow a lot) compared with more traditional CS students.
There were two questions in the survey:
- Under what circumstances do you want to use Stack Overflow? Hint: think at different software development phases.
- When does Stack Overflow fail to fulfill your requirement? Why does it fail?
I collected 67 valid responses, and the response rate is 100% since it’s homework :). Here are my interpreted results:
To the first question:
- The dominant use case is when they encountered confusing error messages and wanted Stack Overflow to help them debug.
- Other use cases can be covered by the taxonomy in this paper.
To the second question:
- The current answer metrics may be not good enough. Some students reported that their legitmate answers were downvoted, while the low barrier of answering makes decent answer hard to discover.
- Knowledge is scattered and hidden in the forum. Current tagging system can’t deeply index such knowledge. One of the many ramifications is that oftentimes you find yourself reading similar questions one by one, and only after ten-something times do you find the answer.
- Answers may be wrong, dated, or violate best practice.
- Some answers were workarounds, not real solutions.
- Some answers didn’t explain. Either they didn’t explain why the problem happened, or they didn’t explain why the solution could work, or both.
- Non-native speakers’ brittle knowledge of English could lead to misunderstanding, which further leads to new bugs and puzzles.
- Good answers can be relative due to different constraints in different projects.
- If you are asking ‘How to do X’ question, you often have to pick the right wording, which is difficult for novices. This may be an eternal pitfall of text-based search.
- Many frameworks and libraries will have the same vague error message in different situations, which not only forces you to look for help, but also leads you to find unfitting answers, which are only correct in other settings.
In hindsight, I should have asked them to provide concrete examples.
Many of these claims are interesting and I never saw them in previous studies on Stack Overflow. Such result demonstrates a wide spectrum of how things can go wrong, and I believe there is a huge opportunity for researchers to dig beneath the surface.