Detailed guidelines


For each task, you will be shown a piece of text comprising a subtitle line from movies.
You will also be shown at most five images.

You task is to select all images suitable for illustrating the given text (or any part of the text).


An image can be marked as either correct, partially correct, or incorrect.
Click on the image to toggle between the three (try it now!)


For an image to be considered correct:

  1. the image (or some parts of the image) should depict the text (or any part of the text);
  2. the remaining elements in the image should not contradict the entire text.

For example, the second image below is correct: it fulfils condition one ("a red car" describes the red car in the image), and the entire text does not contradict any remaining parts of the image (the trees and fence are not mentioned, and the image does not contradict "a black interior" since we cannot tell the colour of the interior of the car in the image)

An image is considered partially correct if condition 1 above is fulfilled but condition 2 is not.
In other words, if part of the image that depicts any part of the text (or the whole text), but some elements in the image contradicts the whole text, then it should be marked as partially correct.

For example, the fourth image above is partially correct. While it depicts a red car (condition 1), the seats are not black (contradicting the text that requires the red car to have a black interior (condition 2 not fulfilled).

All remaining images where the text does not describe any element in the image should be marked as incorrect.

For example, the planes in the example below are incorrect as they do not depict the text.

As another example, images 1 and 2 below are correct because they both depict an upper-leg contusion. Image 4 is correct because it depicts a contusion, and it does not contradict "upper-leg contusion" as the location of the contusion is unclear from the image. Image 3 is partially correct because it depicts a contusion but contradicts the "upper-leg" portion of the text as it is a contusion on the wrist. Image 5 is also partially correct because although it depicts a contusion, it is on the lower-leg rather than the upper-leg.

If an image has words/text superimposed into or embedded within it, such superimposed/embedded text should not be used to inform your decision. That is, the image itself (excluding any superimposed or embedded text) should on its own be able to depict the given text. Otherwise, such cases should be marked as incorrect. If the text occur naturally as part of the image (e.g. a logo on a shirt or a road sign), however, this may be considered as correct.

For example, all images below are incorrect (except the first image).

There may be cases where the text might not be able to be depict really not be able to find any images (even outside the five presented). Please indicate such cases by ticking the checkbox. Note that the checkbox can be ticked if ANY of the statements is true. Naturally, you are not allowed to select any images as correct/partially correct if this checkbox is selected.

Please do not worry about incomplete or ungrammatical text, or text containing strange symbols like $ or |.

Click on the "Save & Next!" button to save your annotation. Please note that you will not be able to return to edit your annotation once this is done.

Remember to click the "Logout" button (top right of page) when you are done or are not intending to annotate for a while.