"We're seeing more issues..." -many Pragli users
It was obvious that we needed to do something. Our users were getting increasingly annoyed at the bugs that were constantly popping up. We were moving fast and shipping a lot of features - but in the process, also breaking things.
Ultimately, we decided we needed a fresh pairs of eyes to look over our product for bugs. In the post, I'll go through our process of hiring a part-time QA tester for $40 / session via UpWork.
We had a pretty clear vision of what we wanted this person to do for us.
We'd made a spreadsheet of all of the manual test cases in our product, and we wanted them to run these tests in various environments, including web (Chrome / Firefox) and our desktop application, on various OSs (Windows / macOS / eventually Linux).
Note: you can look at / take a copy of the sheet here
We had some additional constraints:
- Frequency of 2 weeks: we ship Pragli roughly every other week to balance new features with stability, so we wanted our QA person to execute all of these tests once every other week as well. We figured it would take ~2 hours to execute all of the tests.
- Low budget: as of this writing, Pragli is bootstrapped & in beta (pre-revenue), so we run our business as efficiently as possible.
- Pacific Time: we wanted someone who would run these tests during Pacific Time so that we could synchronously collaborate with them about bugs
We had no clue how to use UpWork. We started by browsing around, searching "QA" on UpWork's "Find Freelancers & Agencies" search.
This gave us a sense of the market
- In the US, prices ranged from $20 / hr to $100 / hr
- Internationally, prices ranged from $8 / hr to $40 / hr
It turned out that you can't just make an offer to hire someone on UpWork. You need to post a job, and then you can invite people to look at the job. Once you receive some applications for your job, you can choose who to hire. Importantly, the job needs to have a price and details about the role.
Even with UpWork's provided metrics like "100% Job Success" and "$100k+ earned," I was pretty nervous about just hiring someone for a job. What if they were terrible?
I ultimately decided that, assuming I got at least three applications, I'd hire three people, each for one session, and then move forward with my favorite on a longer-term basis.
I also wasn't sure where in the relatively high range of prices to put our job price, but I ended up settling on $40 per 2 hour session.
I titled my post "Need regular software QA tester ~$40 / 2 hrs" and then hammered out the following description:
I am looking for a QA tester to manually execute a set of test plans once every other week. I am also looking for the tester to be available to do one additional session in between if necessary.
We are looking to pay $40 per session - each session is expected to take ~2 hours. The first session(s) may take longer while they get setup and acclimate to our testing methodology.
We ask that the tester work do their session once per week at the same time. That time may be Thursday between 3-7pm PST or Friday between 8am-4pm PST to collaborate with the development team when necessary.
Pragli is a virtual office for remote workers. We are available in Firefox, Chrome, or as native Mac and Windows apps. The tester is therefore expected to have a Mac and a Windows computer.
We will happily promote our experience with the freelancer on our blog if our experience is positive.
Here's what the posting looked like:
I then invited a few freelancers that looked relevant and waited for proposals.
The responses were actually overwhelming with over 30 responses within the first 12 hours. Some of them didn't have a lot of experience, others had problems with the listing, but many were qualified and looked great on paper.
We ended up choosing three who had a 99+% Job Success and at least $20k earned in UpWork. To protect their privacy, I'll describe them as Person A, B, and C.
- $20k+ earned, 100% Job Success Rate
- Based in Ukraine
- 7+ years experience in QA
- Fluent in English
- $200k+ earned, 99% Job Success Rate
- Based in India
- 15+ years experience in QA
- Native English speaker
- $20k+ earned, 100% Job Success Rate
- Based in Pakistan
- 6 years experience in QA
- Fluent in English
I confirmed that they fully understood the requirements and then scheduled each of them two weeks apart for their first session (A, then B, then C).
For each of the trial sessions, I scheduled an initial 3 hour video conferencing meeting. I explained to them that in the future it should only take 2 hours without the onboarding. The compensation was still $40.
The agenda was:
- Go over Pragli (demo)
- Add the tester to our Pragli, Slack, and Asana
- Go over our bug reporting expectations
- Overview of test cases
- Run through test cases & find bugs
After introductions, we spent an hour going through the product walkthrough. She asked a lot of questions, which I took as a good sign, though I was a bit concerned about burning through too much of our time. Then, we spent another 30 minutes adding her to our systems and explaining where we wanted her to put bug reports.
Afterwards, she jumped into testing, and in 90 minutes finished our test spec and logged 6 bugs.
Person A really stood out because of how professional her bugs looked. She took the time to identify the version of the app, exact steps to reproduce, expected result, and actual result.
After this session, I felt pretty good knowing that I'd found someone who would do a great job. I even considered canceling the rest, but figured I'd go through them all to see if anyone could do even better.
With Person B, there were immediately more issues. For one thing, she asked almost no questions. Also, while she spoke English well, the audio quality was bad and I had trouble understanding her.
After about 30 minutes, we had gone through what took 90 minutes with Person A, and she asked to be left to do the work. So I got off the call, and I went to work on other tasks, hoping she would blow me away with bug reports.
Two hours later, I received a message from her that she was done, but I noticed that there were no tasks in Asana. After asking if she found any bugs, she said she hadn’t found any issues.
Our product definitely had some bugs, especially the development version we'd sent her. I said thank you and crossed her off the list.
Person C was very enthusiastic and loved learning about our product. He spent a long time trying to understand the ins and outs, and then spent an even longer time finding bugs. Unfortunately, there were some mild audio issues during the call portion. Afterward about an hour, we got off the call and he went to work.
Two hours after our session was scheduled to end, he was still logging bugs. He ended up logging 11 of them in total. Wow! This was going to make it a tough decision. His bugs weren't quite as professional as Person A's, but they were enough to figure out the issue and work on a fix.
Afterwards, he mentioned that he thought that 2 hours wouldn't be enough time to adequately test the app to our specification. He would want more time in the future. I was scared about this... it took us about two hours, but maybe we weren't as thorough as these QA professionals? I thanked him and told him we'd get back to him shortly.
Person C made it a tough decision, but we ended up going with Person A. We first confirmed with her that she thought 2 hours was enough time (she did) and then booked her next session.
Ultimately, she had been a bit easier to communicate with (no audio issues), her bug reports had been a bit more clear & professional, and she felt more comfortable doing quality work in the specified amount of time.
That being said, we still may use Person C in the future as a second pair of eyes on new features. He did a really good job.
We've since had four sessions with Person A, and she's done great.
She starts working around 8am Pacific every other Thursday. She doesn't always do the 2 hours all at once since she has young children, but she always finishes the work during Pacific time on that day, so she can collaborate with us if needed.
She's logged at least 2 new bugs every time, typically more like 4-5, and she also re-identifies all the old bugs that we haven't fixed yet.
Our product is significantly more stable as a result of her work, and we're very happy with our decision.
If you're interested in working with Person A, her name is Kateryna Molina, and you can find her UpWork profile here.
After this experience with UpWork, I'd be significantly more likely to use it in the future, and I think founders should give it a try as long as they go in with the correct process.
Our product, Pragli, is a virtual office for remote workers. Pragli is designed to help teammates communicate faster and build closeness with each other. If you're interested in learning more, click here.