Recently I came across a few articles on the web - people writing about process in general, or about their own (nightmarish) experiences.
The thing that tipped me to actually write this post was somebody writing about the interview for Amazon. I read this and "WTF" would be a rather mild description of what was going in my head. I actually admire this person's patience as I don't think I'd go beyond the "please give me full control over your machine", and most definitely not past the "take all your written materials off your desk, and you're not allowed to use pen and paper". The privacy aspect of this experience is one thing, covered there quite well, and I'm not going to write about it here. In my world it would be unacceptable, period. But what else came to me was the reason for why Amazon considered such extensive control necessary.
Before I go any further with this let's check if we agree on one primary thing. What is the purpose of an interview? Is it
a) to show the interviewer's superiority?
b) test someone's IQ?
c) check resilience to stress and/or ability to work in unknown circumstances?
or
d) check that someone can perform the job that we're hiring for?
Now, my answer to this is most definitely d), and obviously anybody selecting other option will most likely not agree with the rest of reasoning here, so please feel free to drop off the call now.
Quick show of hands. Do you think employees in Amazon (or any other company doing similar tests) during their daily job can:
a) access the internet (you know, google, wikipedia, stackoverflow, the lot...)
b) use pen and paper?
c) use any other written materials, such as books, personal notes etc.?
Why is it then that suddenly, just because it's "an interview for a job", all of that becomes forbidden? How closely does this mimic actual performance?
I mean, seriously. Imagine for a moment that you have to spend a week in your current role and internet access, all written materials AND your personal notepad were taken away. How productive would you be?
Being a good developer is not (just) about knowledge. It's not about how much you remember. How many books you can almost quote from the top of your head. It's about how you apply the knowledge in practice, and how well you manage to deal with things that maybe you haven't come across before.
We're no longer school kids. The testing surely can move on from the idea "unless it's in your head it doesn't exist". A vast majority of what we do is how quickly, how efficiently, how confidently we can use the resources at our hands. When I go to google with the problem - how good my query is. How quickly can I filter out the unrelated stuff from what's useful. How well can I translate solution in SO to my particular circumstances. How well I understand the root cause of the problem (rather than just being able to copy-paste some duct tape solution around it). Am I able to isolate the problem in a more complex situation. And so on and so on and so on. Why is it suddenly considered "cheating" if I do what I always do?
Software development is now so complex, so vast an area that it really is impossible for anybody to remember it all. So why try? Why make the test focus on our "in memory resources"? Yes, I know, the problems that are given are meant to be simple enough that you shouldn't have to use external resources (if that actually is the case is a topic for a different conversation). But that only tests that the candidate can use what's in his head. Where is your test that they can efficiently use what's out there - which no doubt they will have to do on a day to day basis.
Ironically, I think that taking away the internet access tests LESS for how well a person is prepared for the job, than if you allow everything and just watch: how does this person use the power given to them. If I got a pound every time I have to send a colleague to lmgtfy.com I'd be... well, maybe not rich :), but it certainly would pay for a pint or two.
It always reminds me of one exam we had during my university times. Unlike most other professors, this chap allowed us to bring absolutely anything and everything to the exam - books, notes, slides from lectures. Anything you thought was useful. (Communication between students, thus internet, wasn't allowed, but that's rather obvious). You'd think that everybody came out of this exam with straight A but that was definitely not the case. In fact, it was probably one of more difficult ones, a lot of people still failed - but I absolutely loved it. It was timed, and you needed to have an idea what you were doing - but if you forgot this or that detail, you could just look it up. It was the essence of testing how you use your knowledge.
It also reminds me of a fairly simple coding exercise we send to candidates for our India team. Knowing what you're doing it would take you 20-30mins to write it all, tested on colleagues at work. They can complete it at a time of their choosing, spend as much time as they want, use any resources they want. You'd think it's a pointless test and we'd only get perfect solutions. Actual statistics?
- probably 30-40% that doesn't work at all - doesn't compile, is not sent in a project (mvn/intellij or eclipse) format despite explicit instructions, doesn't correctly implement the requirements, is completely not thread safe
- another this many which sort of work but they have a list of "sins" - like no unit tests (again, despite explicit instruction they need to be provided), "tests" that are not tests with assertions but main methods, using strange language constructs/coding style, or effectively single-threaded solutions even though instructions call for "efficient" solution and stress out the multi-threaded aspect of the task. Even asking "make sure you don't send any binaries" (because our company firewall blocks such attachments) is too much to ask
- that leaves us with maybe 10-20% solutions where there is any point talking to the candidate, at which point a couple of "why did you solve it this way" questions reveal whether they wrote it themselves.
I honestly don't understand the obsession with "no internet because it's cheating" coming from anybody who works in this industry. It's not cheating. It's part of my job to know how to use the internet well.
Does it make it more difficult to prepare a task that cannot be copy-pasted and tells you nothing? Well, of course it does. You can no longer ask the candidate to implement "check if number is a prime". But going for "ban usual coding resources" just because you're lazy is no excuse.
So sorry Amazon (and Facebook, and Google, and...). You will not see my CV on that pile you get every day.
The thing that tipped me to actually write this post was somebody writing about the interview for Amazon. I read this and "WTF" would be a rather mild description of what was going in my head. I actually admire this person's patience as I don't think I'd go beyond the "please give me full control over your machine", and most definitely not past the "take all your written materials off your desk, and you're not allowed to use pen and paper". The privacy aspect of this experience is one thing, covered there quite well, and I'm not going to write about it here. In my world it would be unacceptable, period. But what else came to me was the reason for why Amazon considered such extensive control necessary.
Before I go any further with this let's check if we agree on one primary thing. What is the purpose of an interview? Is it
a) to show the interviewer's superiority?
b) test someone's IQ?
c) check resilience to stress and/or ability to work in unknown circumstances?
or
d) check that someone can perform the job that we're hiring for?
Now, my answer to this is most definitely d), and obviously anybody selecting other option will most likely not agree with the rest of reasoning here, so please feel free to drop off the call now.
Quick show of hands. Do you think employees in Amazon (or any other company doing similar tests) during their daily job can:
a) access the internet (you know, google, wikipedia, stackoverflow, the lot...)
b) use pen and paper?
c) use any other written materials, such as books, personal notes etc.?
Why is it then that suddenly, just because it's "an interview for a job", all of that becomes forbidden? How closely does this mimic actual performance?
I mean, seriously. Imagine for a moment that you have to spend a week in your current role and internet access, all written materials AND your personal notepad were taken away. How productive would you be?
Being a good developer is not (just) about knowledge. It's not about how much you remember. How many books you can almost quote from the top of your head. It's about how you apply the knowledge in practice, and how well you manage to deal with things that maybe you haven't come across before.
We're no longer school kids. The testing surely can move on from the idea "unless it's in your head it doesn't exist". A vast majority of what we do is how quickly, how efficiently, how confidently we can use the resources at our hands. When I go to google with the problem - how good my query is. How quickly can I filter out the unrelated stuff from what's useful. How well can I translate solution in SO to my particular circumstances. How well I understand the root cause of the problem (rather than just being able to copy-paste some duct tape solution around it). Am I able to isolate the problem in a more complex situation. And so on and so on and so on. Why is it suddenly considered "cheating" if I do what I always do?
Software development is now so complex, so vast an area that it really is impossible for anybody to remember it all. So why try? Why make the test focus on our "in memory resources"? Yes, I know, the problems that are given are meant to be simple enough that you shouldn't have to use external resources (if that actually is the case is a topic for a different conversation). But that only tests that the candidate can use what's in his head. Where is your test that they can efficiently use what's out there - which no doubt they will have to do on a day to day basis.
Ironically, I think that taking away the internet access tests LESS for how well a person is prepared for the job, than if you allow everything and just watch: how does this person use the power given to them. If I got a pound every time I have to send a colleague to lmgtfy.com I'd be... well, maybe not rich :), but it certainly would pay for a pint or two.
It always reminds me of one exam we had during my university times. Unlike most other professors, this chap allowed us to bring absolutely anything and everything to the exam - books, notes, slides from lectures. Anything you thought was useful. (Communication between students, thus internet, wasn't allowed, but that's rather obvious). You'd think that everybody came out of this exam with straight A but that was definitely not the case. In fact, it was probably one of more difficult ones, a lot of people still failed - but I absolutely loved it. It was timed, and you needed to have an idea what you were doing - but if you forgot this or that detail, you could just look it up. It was the essence of testing how you use your knowledge.
It also reminds me of a fairly simple coding exercise we send to candidates for our India team. Knowing what you're doing it would take you 20-30mins to write it all, tested on colleagues at work. They can complete it at a time of their choosing, spend as much time as they want, use any resources they want. You'd think it's a pointless test and we'd only get perfect solutions. Actual statistics?
- probably 30-40% that doesn't work at all - doesn't compile, is not sent in a project (mvn/intellij or eclipse) format despite explicit instructions, doesn't correctly implement the requirements, is completely not thread safe
- another this many which sort of work but they have a list of "sins" - like no unit tests (again, despite explicit instruction they need to be provided), "tests" that are not tests with assertions but main methods, using strange language constructs/coding style, or effectively single-threaded solutions even though instructions call for "efficient" solution and stress out the multi-threaded aspect of the task. Even asking "make sure you don't send any binaries" (because our company firewall blocks such attachments) is too much to ask
- that leaves us with maybe 10-20% solutions where there is any point talking to the candidate, at which point a couple of "why did you solve it this way" questions reveal whether they wrote it themselves.
I honestly don't understand the obsession with "no internet because it's cheating" coming from anybody who works in this industry. It's not cheating. It's part of my job to know how to use the internet well.
Does it make it more difficult to prepare a task that cannot be copy-pasted and tells you nothing? Well, of course it does. You can no longer ask the candidate to implement "check if number is a prime". But going for "ban usual coding resources" just because you're lazy is no excuse.
So sorry Amazon (and Facebook, and Google, and...). You will not see my CV on that pile you get every day.
