Top-down Processing in Visual Perception Part IV: Ramifications

This is the final instalment of my series on top-down processing in the visual system (links to part I introducing the topic, part II discussing faces and anthropomorphizing, and part III discussing artificial edges). While I find the topics of vision and optical illusions to be fascinating in their own right, I think the analysis of perception and cognition is also vitally important. This is by no means an original outlook, as David Hume made the statement in the introduction to his A Treatise of Human Nature:

'Tis evident, that all the sciences have a relation, greater or less, to human nature; and that however wide any of them may seem to run from it, they still return back by one passage or another. Even Mathematics, Natural Philosophy, and Natural Religion, are in some measure dependent on the science of Man; since they lie under the cognizance of men, and are judged by their powers and faculties.

While couched in somewhat archaic English, Hume's statement strikes me as remarkably astute. In many ways, our brains function as vast pattern-matchers. Understanding the underlying cognitive tricks we use to analyse perception is an important endeavour for making sense of our own observations, and avoiding mistakes in our interpretation of experimental results. Of course, the most pertinent application of perceptual understanding is in automated sensory processing applications (like machine vision which I have discussed before), but as Hume pointed out, it also matters in the way our thought processes interact with every other endeavour. We must be wary of our tendency to anthropomorphize, or to view causal connections that are not actually there. Realising our tendency to perform processing without being consciously aware of it helps reinforce the necessity of mathematical, logical, and statistical tools on which to rest one's theories.

Some more evidence that machine vision is hard

Nearly a year ago, I made a post talking about the woefully difficult task of visual processing, and it is a theme I have continued to discuss in my series on top-down visual processing, particularly the third (and most recent) instalment. Well, now it is time to take a sojourn out of the ivory tower of academic discussion (a phrase which I, of course, use ironically) and giggle at some real-life examples of failed automated vision tasks involving iPhoto's facial recognition software. The examples have all been submitted to a Flickr group where they can be perused at one's leisure.

I think this one is my favourite:

Top-down Processing in Visual Perception Part III: Artificial Edges

This was supposed to be the concluding chapter in my series on top-down visual processing started in part one and continued in part two, but it got quite a bit longer than I expected and will thus be expanded in an upcoming fourth part. In the first installment I introduced the concept of top-down and bottom-up processing and gave a low-level example of top-down processing in the Necker Cube. In the second part I discussed faces in the context both of anthropomorphizing objects through the visualization of faces as well as a preponderance of optical illusions involving faces.

In this installment I am going to discuss another artifact of top-down visual processing which I am going to call artificial edges (I'm not actually sure if there is a better or more technical term for it, so if anyone knows of one, feel free to let me know). I find this phenomenon interesting from both a physiological aspect (in terms of providing evidence for top-down processing) as well as a machine vision aspect (in terms of duplicating our object recognition abilities). The basic idea is that our brain is fairly good at joining edges which belong to the same object but which have been in some manner obscured (either through occlusion, camouflage, or illumination problems). What is interesting, however, is that our visual processing system is so good at this that we can actually create edges and object boundaries that are not there. Two classic examples of this are shown in figures 1 and 2.

Figure 1

Figure 2

There is not actually a white square in figure 1, but the pieces removed from the black circles give the illusion of a white shape occluding them. Our mind then fills in the boundaries of the square to separate it from the white background to the point where we are able to discern boundary lines that are not actually there (of course, those lines disappear when one focuses on them since there isn't actually a change in hue). Figure 2 shows a somewhat more complicated version of the same phenomenon in which the image of a dalmatian is hidden within the scattered ink blots. The artificial edges created in these images are a consequence of our ability to group objects together and differentiate foreground from background. Part of our ability to do this rests in our expectations of what objects are most likely to appear in an image as well as our expectation of how such objects might be arranged. If figure 2 were shown to an individual who had never seen a dog (or even a dalmatian or similarly coloured dog), he would most likely have a very difficult (if not impossible) time spotting it.

That said, I am going to make a brief digression and point out one of the things that makes psychology such an annoying subject - differences in individual processing. Just as some people mentioned they had a difficult time spotting the old lady in the previous part, my girlfriend told me she does not see four circles occluded by a white square in the first image, but rather she sees four Pac-men biting a square. At the least the square is still there so she doesn't completely spoil my premise.

This ability to group contours and blobs together into expected objects is a massive advantage when it comes to image understanding, and it is one of the biggest problems in machine vision. Outside of tightly controlled circumstances, object contours rarely display consistent properties. This is hard for people to even spot, because our mind automatically accentuates valid contours and minimizes invalid contours as shading and texture.

Figure 3

To demonstrate this, I have included a picture I pulled off the internet of a chrome stapler (figure 3). As one can see, there are plenty of strong and weak edges in this image (when I speak of the strength of an edge, I mean roughly the rate at which pixel intensity changes. For a more thorough explanation, see the Wikipedia article). People have no difficulty picking out the stapler in this image and could easily outline the object if one were to ask, despite the fact that this is a monochromatic image and several of the boundary edges are much weaker than internal edges caused by shadow and geometric variations in the object's surface. For example, if you look at the two rearmost edges of the stapler, you can see that the posterior edge is virtually nonexistent while the lateral edge starts fairly strong near the bulb at the front of the stapler but fades as one moves toward the posterior. Our minds have no problem mentally accentuating that lateral edge along its entire length, however, and recognizing that it is a continuous edge despite its vast variation in edge strength. If you looks at the opposite side, however, you see a continuous dark band that extends the length of the stapler, forming two powerful edges. Neither of these strong edges actually depicts one of the object's boundaries, rather they are an artifact of the object's geometry, the view angle, and lighting. Thus, even a computer system whose sole purpose is to determine if one has a picture of a stapler or not would have a great deal of difficulty with that task without some pretty hefty processing on top of the edge detection (even then it would highly unlikely to be as reliable as a person, and we can recognize far more objects than just staplers) or some ability to constrain the view angle, lighting, and object variation. A great deal of these concepts are actually discussed in Gestalt psychology (if you follow the link to the Wikipedia article, you will see some familar images too. It looks like I could have just acquired links from there rather than searching randomly through the internet if I had looked at Wikipedia earlier).

I had planned on speaking about the ramifications of what I have discussed, but I have already been working on this post on and off for several weeks and it is starting to get cumbersome in length. I will therefore be expanding this series into a fourth post to be published in the not too distant future.

Continue reading in Part IV: Ramifications.

Top-down Processing in Visual Perception Part I: Introduction and Some Examples

One of the subjects I have written about before is machine vision and the incredible difficulty of developing a robust visual processing system that can equal the robustness of our own visual system. It shouldn't be entirely surprising, though, that our visual system is as incredibly powerful as it is, since a huge proportion of our brain is utilized primarily for visual processing. One of the interesting debates in perception psychology and neuroscience is whether the brain performs bottom-up or top-down processing. As with most things (especially in psychology), neither one is entirely correct and your brain utilizes a combination of the two. Optical illusions and trick images are one relatively simple way to explore the way our brain processes visual information, and they are also fairly fun to look at.

Bottom-up processing basically means your brain reads in the raw visual information captured by the retina and gradually figures out what it means as one moves farther along the processing chain that is your cerebral cortex. Top-down processing means you start with an idea of what you ought to be seeing (most likely determined by recent sensory information, other sensory clues, and your past experience). Your brain clearly does some bottom-up processing, since you react to raw changes in the visual stimuli even if there was no reason to expect that change. What is fairly surprising, though, is top-down processing is also clearly involved in visual processing. Effectively introducing top-down processing into artificial visual systems, however, is quite difficult, and it would seem that the top-down algorithms instituted by our brains (and their handy parallel architecture) are what keep us currently so far ahead of computers.

One example of top-down processing that is fairly easy to demonstrate is the blind spot. In your retina you have a small area devoid of receptors where nerves and blood vessels enter and leave your eye. This is normally not a problem since the blindspot of each eye falls on a different area of your visual field, so the sensory perceptions of one eye can compensate for the other. Also, your eyes are almost constantly performing saccades (small jumps around to focus on different regions of the visual field). However, if you close one eye and keep your other eye locked on a specific target, your blind spot becomes anchored in place. You do not realise this, though, because your brian manages to fill in that area of your visual field with its best guess as to what is there. A quick way to demonstrate this is to take a piece of scrap paper and put two X's on it about eight centimeters apart. Then close one of your eyes and stare at the opposite mark with your open eye (for example, if you closed your left eye, look at the left X with your right eye). Hold the paper about half an arm's length in front of you and gradually move it closer. At a certain point, the X on the periphery of your vision should disappear. When it does, it is sitting in your blind spot, and your brain fills in that area with it's best guess (in this case, blank white paper).

Another example that occurs slightly higher up in your visual processing is the Necker Cube, shown here.

This simple drawing forms a three dimensional clear cube. It is ambiguous, though, whether it is intended to be in one of two possible orientations: are you looking slightly down onto the cube, or slightly up at it (in other words, are the bottom two corners corners on the front or back face of the cube)? For most people, there is a default orientation when they first see it. However, after staring at the cube for a few moments, they can cause it to 'flip' into the other orientation. At no point, though, can both orientations be held in one's head at once (at least, I cannot manage to do that). It would seem that your brain takes the visual information provided about the cube's edges and then tries to fit an interpretation on it. Since more than one interpretation is possible, your brain alternates between them. However, whenever one particular interpretation is selected, the others are suppressed to avoid conflicting interpretations of a visual scene.

Continue reading in Part II: Faces.

Scientist Appreciation: Sven Dickinson

The scientist for this week is actually one of my professors, Dr. Sven Dickinson. He taught the Image Understanding course I took this past spring, which is a specific area of machine vision involved in the most unconstrained vision problems (many objects with many different orientations and articulations). The reason I feel he should be should be appreciated is because he is very much the sort of professor I would like to be. He is extremely engaging while at the same time is very approachable without an overbearing aura of intimidation. In his lectures he incorporates a wide range of subject matter to help give an impression of how the field fits into scientific endeavours as a whole, as well as helping students realise that multidisciplinary knowledge, even just superficial knowledge, can provide insights that otherwise would have been missed. In addition to his impressive abilities for delivering lectures, he also gives the strong impression that he genuinely cares about the students in his course, both in terms of their understanding of his course as well as their overall well-being and future goals. He takes time to offer advice on a wide variety of subjects, and fairly adjusts schedules and course work in the light of unforeseen difficulties.

He is an impressive scientist not just in his role as a teacher of science, however. He has a broad view of his field that allows him to see the trends that come and go without getting swept up in the short-sighted the frenzy of adoration that can sometimes captivate large groups of researchers over the newest method or algorithm that shows any promise. It is professors and researchers like Dr. Dickinson who should be sought out and listened to if we are to effectively support and guide future scientific research.

Vision is Hard

Vision is a remarkably difficult problem. One of my favourite "trick" images that demonstrates some of the challenging aspects of vision is the one displayed above. In the image, there are two squares labelled A and B. Those squares are identical in colour. It takes a bit of effort for most people to accept that, however. If you don't believe me, try loading the image into an image editing program and either get colour information from each square, or copy and paste regions of each next to each other.

Once satisfied that the two squares are actually identical in colour, it is worth stopping for a moment and thinking about why they look so remarkably different. In my experience, most peoples' initial reactions to this image are "how could our eyes screw up so badly?" However, if you think about it, it could equally be viewed a mistake to see the two squares as being identical in colour. It all comes down to the job one wishes to do. In most complex vision tasks, however, grasping the overall pattern of the scene is quite important, which makes viewing the two squares as different shades the more desirable perception to have. However, attaining the perception that a human effortlessly has (in fact, it takes a fair bit of effort to overcome the apparent difference in magnitude of grey) is actually quite difficult for a computer.

Say, for example, that you wish to take an image of a checkerboard like the one in the above image, find all the exposed squares, and label each square as either a dark or a light square. Since we are allowing the possibility of occlusion, variable illumination, and shadows, clearly there can be no simple global threshold used to label each pixel as either a dark or light pixel and then simply group like pixels with each other. If one attempted to do that, A and B would be labeled in the same manner and, therefore, one would be incorrect. So, perhaps you decide to be a little more complex and a little cleverer than that. You decide to locate the top left corner of your checkerboard (assuming that it is never occluded, which can sometimes be a hefty assumption, but we'll allow it) and then find its boundaries using either a blob tool (group all nearby pixels within a certain range of the starting pixel's value) or edge finder. Then, once all the boundaries have been located, compare neighbouring square average pixel values to determine which ones are the light and which ones the dark squares. Unfortunately, there are several problems with this approach. The first is the possibility of falsely splitting a square because of a shadow lying across only part of it. The second is that the thresholds used to find the boundaries of each square will not work at all illumination levels. Algorithms do exist for trying to dynamically find appropriate thresholds, but they are beyond the scope of this brief discussion and are not completely reliable.

Thus, you might decide to try one last method of being to clever to fix the downfalls of your previous method. You model the size of the squares with predetermined values, so once you find one you will be able to know where the others are even if the edge or blob detector fails to find a boundary or finds an extra boundary. However, this will only work if the orientation and apparent size of the board are completely fixed, which is not a reasonable assumption for any but the most constrained environments.

This is not an impossible task. It is not even a particularly difficult vision task if one were able to constrain the position and illumination of the checkerboard. Constraining at least one would still make it much simpler. However, I hope this simple discussion has made it clear just how much more complicated even a simple task like this might be, especially once the light levels and orientation are allowed to vary. When the task is scaled up to allow a wide assortment of objects, it becomes virtually intractable. I say virtually intractable because it clearly must not be, since it is a task that nearly every person on the planet manages to accomplish every day. Figuring out just how that is done, though, is a very fun problem to cogitate.

Scientist Appreciation: David Lowe

Oddly enough, the scientist for this week doesn't have a Wikipedia page. While there are plenty of scientists without Wikipedia pages, this fellow was repeatedly mentioned in two of my courses this semester with several ground-breaking computer vision algorithms to his name. The ubiquity of his name in my courses while at the same time his relative anonymity online I find slightly odd (though, of course, he has his own website from UBC where he currently teaches).

The algorithm of his that I am the most familiar with is called SIFT (scale-invariant feature transform, which does have a Wikipedia page), which is a remarkably robust image recognition algorithm for finding specific objects in images. The actual difficulty of this problem is a great deal higher than most people realise (a subject that I might write about in the near future), and the accuracy of the SIFT algorithm is quite impressive. In addition to SIFT and other recognition work, Dr. Lowe has also done work on automatically stitching together panoramic image scenes from a disjointed set of images, robot guidance, and scene manipulation.

While his list of accomplishments and influential algorithms is quite long and impressive, what is also great about David Lowe is that he is still conducting research at UBC. This means that there is actually a small iota of a chance that I might actually get to meet him one of these days and make some sort of silly, awestruck comment about how remarkably clever his work is.