We want to capture a simple concept that can sometimes manifest itself in very complicated ways. In particular, we ask the following question:
“Why did the photographer take this particular picture?”
This is not a question of artistic composition, however. Pictures should be contributed to Wildbook for the purposes of identifying individual animals. Therefore,
to answer the "why" question, the goal is to understand an image's semantic composition by marking the individual annotation(s) that are the
interest,
subject(s) or
focus of the picture. More specifically, the task is to mark all annotations that have (in general) the following properties:
- Is of a distinguishable individual (i.e. free-standing or clearly in the foreground of a heard)
- Is relatively large and has decent resolution
- Is commonly located near the center of the image
- Is in focus and not blurry
Conversely, the annotations that are
not considered to be an AoI have (in general) the following opposite properties:
- Is a part of an overlapping herd or group of animals
- Is relatively small and/or contains few pixels
- Is out of focus and is blurry (from being too far from or too close to the camera relative to the plane that is in focus)
- Is located around the edges of the image
- Is significantly (greater than 25%) occluded by other animals or objects
- Is significantly (greater than 25%) off the edge of the frame of the image
These above properties (and counter properties) require that the annotation must not be viewed in isolation. The decision that an annotation is an AoI must be made by weighing the entire context of the image and against any other annotations that may also be in the image. Further, because these conditions are fairly strict, there are rarely more than a few AoIs in a particular image, and some images may have none.
Note: As a small (but important) aside, all individual animals (of the target species) in the image should have an annotated bounding box. The intent is to have the same number of bounding boxes as the number of animals that can realistically and reliably be manually counted in the image. If one can reasonably count 30 animals in the image, there should exist 30 corresponding bounding boxes. This completeness assumption is crucial to the validity of the Annotation of Interest decision for a given image; any image that does not satisfy this condition should first be reviewed to obtain a complete set of annotations.
Beware: it is crucially important to not accidentally conflate this metric with other detection and identification concepts. While the concepts below are related to and may be strong indicators of Annotation of Interest, they are not necessarily identical. For example,
- Annotation of Interest ≠ is identifiable by any particular algorithm
- Annotation of Interest ≠ is of a desired quality
- Annotation of Interest ≠ is of a desired species
- Annotation of Interest ≠ is of a desired viewpoint
- Annotation of Interest ≠ is free of photo-bombs
How well do these rules work on real-world examples? Let's take a look at a few:
Example 1
First, there is the need to define the color scheme. An annotation bounding box is marked with
orange or
green unless it is an AoI, which is marked with the color
blue.
Tip: Use the hotkey I (for interest) to toggle a hovered box as an AoI
Tip: Sometimes bounding boxes are very close together and overlapping. If you experience difficulty hovering the desired bounding box, use the hotkey B (for background) to send the hovered (undesired) annotation to the background.
This first example (above) is straightforward. There is only one annotation in this image, it is clearly lit, and has no occlusions. A free-standing animal in a field, like in this example, can be considered an ideal AoI. The annotation is therefore marked in blue to signify that it is an AoI.
Remember, not all images are guaranteed to have at least one annotation. Furthermore, for an image with a single annotation, it is not guaranteed that the solo annotation should always be an AoI.
Example 2
This second example gives multiple instances of what is considered to be an out-of-focus annotation.
The single AoI in this image (again, in blue) is smaller by area than in the first example. The blue annotation is in focus and clearly distinguishable, which makes it an AoI for this image. The four additional orange annotations near the top of the image are not AoIs: they are small and out of focus.
Example 3
This example has 4 total annotations, two of which (in blue) are AoIs and two which are just normal annotations (in orange). The two smaller annotations in the background do not qualify as AoIs because they are slightly out of focus, too small, and the left-most orange box has significant occlusion by another animal. The two blue AoIs are both in focus and have no significant occlusions (the small amount of occlusion of the left AoI by the right AoI is not significant nor is the clipping end of the snout by the edge of the frame).
Looking at the AoIs from these first 3 examples, a pattern begins to emerge. This pattern revolves around what can be called the
"focus plane" and it helps to outline a simple procedure for deciding which annotations should be considered an AoI.
- Find the single best AoI in the image, if it exists. It is important that the best AoI is in focus and has the easiest argument for being an AoI. (For example, the right of the two blue bounding boxes in Example 3 should be chosen as the best AoI because the zebra is more central in the image and it slightly occludes the zebra in the left blue bounding box.) If no such AoI exists, continue to the next image without selecting any AoIs.
- Determine the "focus plane" that this best AoI lives within. The focus plane can be decided based on the regions of sharpness and blurriness within the image. This theoretical plane is the depth within the image that the best AoI occupies in 3D space. For example, all other objects that are also within this plane should be all have the same focus and depth within the image as the best AoI. Note that not all focus planes will be the same width; an image taken with a larger camera aperture opening will give a narrower depth of field.
- Any annotation within the focus plane should be considered as a candidate AoI and must be reviewed individually. Any annotation that satisfies the AoI properties and is also within the focus plane of the best AoIs should also be marked as an AoI.
- Any annotation that is not within the focus plane should not be considered an AoI.
A caveat to the procedure above is when the focus plane is very wide (small aperture opening on the camera) and objects at different actual depths in the image can both appear to be similarly in focus and roughly within the same focus plane. In this scenario, these annotations may be treated as if they were in the same focus plane and occupied the same 3D depth within the image.
Let's apply the procedure above to the next example:
Example 4
In this fourth example, the concept of a "focus plane" is demonstrably clear. Let's follow the decision steps just outlined:
- Deciding the single, best AoI in this example is subjective. The most in-focus and free-standing annotations are in the center of the image (marked in blue). For the purpose of discussion, however, we can pick the blue annotation on the right as the best AoI because it is (slightly) more central to the image.
- With the best AoI selected, the depth the focus plane for this image can be decided. This example image can be conceptually broken up into three distinct depth zones relative to the camera: in-front of the focus plane, the focus plane itself, and behind the focus plane. Note that not every image has these three distinct depth zones and some images may only have one. For this example, the bottom 1/3 of the image is in-front of the focus plane, the focus plane occupies the middle 1/3, and the top 1/3 is behind the focus plane.
- With the three focus zones defined, there are three annotations (the two blue and the cutoff orange annotation off the left border) that are within the focus plane. Next, each box is individually checked against the AoI properties. The left blue annotation is a good AoI, but the left-most orange annotation is largely cutoff and should not be considered as an AoI.
- The 4 annotations at the bottom of the image fall outside (in-front) of the focus plane and are therefore should not be considered as AoIs. The 4 annotations at the top of the image also fall outside (behind) of the focus plane and should similarly not be considered as AoIs.
Example 5
Example 5 shows a group of animals in a tight herd. The herd itself is the primary feature of this image rather than any one individual. There are no clearly distinguishable individuals apart from the herd and therefore no AoIs. Also even though all of these annotations occupy roughly the same focus plane, the annotations are also too small to be considered AoIs.
Note: Herds dilute interest. This example's scenario of smaller bounding boxes along with being of a herd is one of the most dangerous in terms of ambiguity and difficulty. Usually these examples will produce no AoI's.
Example 6
The image in Example 6 is striking because there are many annotations, but none should be considered AoIs. This is primarily because the focus plane is clearly centered around the bird stuffed animal. The right-most (blurry) orange annotation would normally be the best AoI, but is ignored for being behind the focus plane.
Example 7
Here is another example that shows many (small) annotations in a herd. None of these annotations should be labeled as AoIs.
Note: Remember that it is crucial for an image to have annotation completeness. To a reasonable effort, every discernible animal in the image should have a corresponding bounding box.
Example 8
With this example we want to reiterate that the decision to label an annotation as an AoI should not be decided by or conflated with other metrics, like viewpoint and pose. The focus plane is fairly wide in this image and also contains all three annotations. The pose of the two lower annotations do not discount them as AoIs.
Example 9
Finishing the instructions, there is an important scenario that needs special attention. If an image contains closely overlapping annnotations of what are
clearly a mother and a foal, then the two annotations should both be marked as AoIs, or both not marked as AoIs. This represents a circumstance where it is quite difficult to obtain clear photographs of each animal individually and also where the association between mother and foal annotations is important for later analysis.
Example 10
This final example shows how complex a mother and foal pair of annotations scenario can present itself. In this example, there are two foals and two corresponding mothers. The two foals can be marked as AoIs and the two mother annotations should therefore be marked as well.
P.S.
We have special instructions for sea turtles. We relax the occlusion percentages from 25% to 50%.
- Is significantly (greater than 50%) occluded by other animals or objects
- Is significantly (greater than 50%) off the edge of the frame of the image
Example 11
Example 12
Examples 11 and 12 shows an AoI of sea turtles. The image of example 11 shows a clearly visible turtle and is not occluded where example 12 shows a turtle that is only partially occluded. Since we can see at least 50% of either animal, we consider both to be AoIs.
Example 13
The next image, example 13, shows a sea turtle that is too occluded to be considered an AoI as it only shows the head of the turtle. The occlusion threshold for sea turtles should be relaxed to 50% compared to 25%.
P.S.S.
We also have special instructions for whale flukes. We relax the occlusion percentages from 25% to 50%.
- Is significantly (greater than 50%) occluded by other animals or objects
- Is significantly (greater than 50%) off the edge of the frame of the image
Example 14
Example 14 shows an AoI of a whale fluke, where the body of the whale is completely occluded by the water but the fluke is not occluded and clearly visible.
Example 15
The next image, example 15, shows a whale fluke that is too occluded to be considered an AoI. The occlusion threshold for whale flukes should be relaxed to 50% compared to 25%. We consider the starting point of the whale fluke to be where the forward part of the fin meets the main tail body.