We've spent a lot of time building a platform that makes it quick and easy for video professionals to upload, share and get feedback on videos during production.
An important aspect of our service has been making sure that we offer fast and reliable video uploading and encoding. When you are on a deadline and trying to get media out to a client for review, you don't want to get bogged down with failed uploads and lengthy encoding queues.
We've seen a complaints on social media about Vimeo's upload speeds, and we've seen a number of people switch from Vimeo to ScreenLight for collaboration because of our performance.
As a result, we thought it's time to put things to the test and compare ScreenLight and Vimeo's uploading and encoding performance. In this test we have focussed on comparing ScreenLight with Vimeo Pro ($199 / year), because it promises faster video conversion than Basic Vimeo Accounts and Vimeo Plus ($49 / year).
Given that videos from Pro account holders jump to the head of the queue, this plan should have the highest performance of Vimeo's offerings.
Total Time Before Video is Ready for Viewing
The best measure of uploading and encoding performance is the total amount of time it takes from moment you are ready to send a video off for review until you can notify your client that it's ready to watch. ScreenLight clearly comes out ahead of Vimeo in this regard.
Uploading and processing a 5-minute test clip (methodology and specs follow) took an average of 70% longer with Vimeo than it did with ScreenLight. Across all five tests with a 5-minute review clip, the average time to upload and encode the video with Vimeo was 21:48 versus 12:48 with ScreenLight.
ScreenLight's slowest total time (14:00) was still better than Vimeo's fastest total time (15:00).
With a 12 minute video encoded with similar specs, it took an average of 36% longer before the video was ready for viewing with Vimeo than with ScreenLight. Across five tests, the average total upload and encode time with this file was 52:00 with Vimeo compared to 38:12 with ScreenLight.
ScreenLight exhibited more consistency in performance as is shown by the lower values for standard deviation of upload and encoding times (a lower value indicates more consistent performance). Consistency is important because it helps you estimate how long it will take before your video will be ready for viewing by a client.
Why is ScreenLight faster than Vimeo?
It turns out that upload times for Vimeo and ScreenLight were very similar. The big difference was time spent processing video. ScreenLight had shorter queues before processing began and it processed video faster than Vimeo.
Upload Times
It took an average of 11:13 to upload the 5-minute test clip to Vimeo, compared with 10:13 to upload the same clip to ScreenLight.
Although this is a 10% difference, and all the uploads to ScreenLight were slightly faster, the difference is not significant, and could be accounted for by variations in the upload speed of our connection, or a difference in our proximity to Vimeo and ScreenLight's respective servers.
When uploading to well tuned cloud services, the upload time is going to be largely determined by your upload bandwidth rather than the service itself. In fact this is what was observed in the tests with the 12-minute video. It took an average of 30:14 to upload the video to Vimeo, compared with 30:13 with ScreenLight.
It seems that with a run of the mill cable Internet connection, advertised as having an upload speed of 2 Mbps, the limiting factor for uploads is the speed of the connection rather than the design of the video sharing service.
Video Processing Times
The real advantage with ScreenLight is that videos are processed for viewing on multiple devices much faster than with Vimeo.
During our testing with the 5-minute clip, it took Vimeo an average of 10:34 to encode the video, versus 2:35 for ScreenLight. In other words, it took almost 4 times as long for Vimeo to process videos for viewing.
With the 15-minute test clip, it took Vimeo an average of 21:45 to encode the video versus 7:58 for ScreenLight. This is nearly 3 times as long.
Vimeo's processing performance exhibited much more variability than ScreenLight's as is shown by the chart below on standard deviation. The 5-minute clip took between 4:47 and 18:15 to encode with Vimeo, as compared with 1:51 to 3:51 for ScreenLight. The 15-minute clip took between 6:15 and 49:34 to encode with Vimeo, as compared with 3:25 to 18:43 with ScreenLight.
There are two factors behind Vimeo's slower processing times. First, videos sit in a queue before they are processed. There seems to be a fair amount of variation in queue times with Vimeo. Sometimes a video will start processing almost immediately, while other times it can take upwards of half an hour.
The second reason for the slower encoding performance with Vimeo is that SD and HD videos appear to be processed serially rather than in parallel.
The problem with Vimeo's variability is that it's difficult to budget for. If you have a client on the other end that's asking you when the video will be ready, you basically have to assume the worst because you don't want to over-promise and under-deliver.
Methodology
We used two different test clips of the movie Tears of Steel to evaluate uploading and encoding performance of Vimeo and ScreenLight over the course of a number of days.
Tears of Steel Clip - The 5 minute clip (H.264 / MP4) has a frame size of 1280x534 and an average bit-rate of 1950 Kbps. Total file size is 73.3 MB.
Tears of Steel Full Version. The 12 minute and 14 second video (H.264 / MP4) has a frame size of 1280x534 and an average bit rate of 2310 Kbps. Total file size is 212 MB.
The 5 minute version was chosen as it's a typical example of the length of videos that people upload for review. The 15 minute version was chosen as it represents times for longer videos and larger files. Dimensions and encoding specs were chosen as they are representative of what we have seen people upload for review using ScreenLight.
Each clip was uploaded 5 times to each service. To minimize the impact of variations in the upstream speed of our Internet connection, we alternated between uploading a video to Vimeo and uploading it to ScreenLight.
The nominal differences in upload times indicate that variations in the speed of the Internet connection did not influence the test results.
We used ScreenFlow to record the screen during the tests. These recordings were used to calculate upload and encode times. For comparative purposes we measured the time until the HD version of a video was available for viewing. In the case of Vimeo, an SD version is made available slightly earlier than the HD version, however, in the case of client review where video quality is important, we believe HD upload time is the best measure of performance.
For the purposes of this test we didn't perform a subjective analysis of encoding quality, although it's an important attribute of any video platform and something that both Vimeo and ScreenLight prioritize.
Bottom Line
In our small scale test we didn't run into the type of failed uploads and major processing delays that have caused people to complain about Vimeo on Twitter. These complaints appear to be the exception rather than the norm.
Nevertheless, it's clear that there is a difference in performance of the two services when it comes to uploading and processing video. While this may not matter for mass distribution when people are typically more concerned with scalability and cost effectiveness, it's fairly important when reviewing video with clients.
The difference in performance comes down to design decisions. Vimeo Pro is built on the same technology stack as their consumer offering. There are some tweaks like priority encoding, but it's not designed from the ground-up for rapidly sharing files with teams and clients.
When we started encoding video for customers back in late 2011, one of our priorities was to optimize our design to keep queue time as short as possible. This is more costly for us than running a service with longer queues (to keep the queue short you have to add more capacity during peak periods of encoding), but it's a decision that's aligned with our customers' needs.
In Vimeo's case, they have chosen to optimize their encoding system to keep costs low. This makes sense given the broad market and extremely low price points they are targeting.
When comparing ScreenLight and Vimeo for reviewing video and getting feedback from clients, faster upload and encode times are just the tip of the iceberg in terms of time savings. This is about more than just reliably getting video to client when there is a deadline. The real time savings come from moving your workflow away from email, centralizing feedback with timecoded comments, and keeping everything for your project organized and in one place, but that's another post...
