[pageinfo chapter=”mind” chapnos=”4″ chaptitle=”Mind”/]
As a physical thing
Our brains are also physical, apparently inert grey goo, yet full of electrical activity running through networked neurons and flooded with neurotransmitters. Three different kinds of physical processes: electrical, structural (connectivity) and chemical working together.
In the 19th Century, many believed that one could tell a person’s character and skills through the shape of the skull, reading ‘bumps’. This study of ‘phrenology” did make sense: if the brain is the seat of character and intellect, and if some parts of your brain are bigger then normal, then surely this should be able to tell you about the person. In fact, we now know that even gross measurements do not correlate well with specific abilities. For example, the total volume of the brain typically varies between individual by up to 50%, but there is no connection between this and intelligence.
(image course: http://en.wikipedia.org/wiki/Phrenology)
However, whilst the raw size of parts of the brain is not related to specific traits or abilities, the physical parts of the brain do perform different jobs; there is functional separation. This was first discovered through accidents involving traumatic injuries to parts of the brain, or through tumours that affected specific parts of the brain. Now-a-days the locations of these would be determined by X-rays or brain scans, but in the early days doctors often had to wait for the patients to die to study the brain during autopsy.
Brain surgeons now use this knowledge to avoid damaging areas associated with critical functions such as Broca’s area and Wernicke’s area, which are essential for different aspects of language. However, the same surgery has also offered fresh knowledge. While the broad structures of the brain are roughly similar between individuals, the details vary and in order to determine precisely the limits of critical areas surgeons will often use a probe to administer tiny electrical impulses to different parts of the exposed brain. The patients, still awake despite having part of their skull removed, perform simple tasks and if the impulse disrupts the task then this tells the surgeon where to avoid, but also reveals more about the fine structure of the brain. For example, stimulating an area may trigger a particular memory of an experience.
Modern brain scanning has allowed scientists to study the patterns of activity of the brain less invasively. One example of this reveals the rich nature of imagination. Subjects are placed in a brain scanner and then shown a picture. The scanner shows the parts of their brain that are active in response including the visual system and the other parts connected with the interpretation of the picture. Then the picture is removed and the subjects are asked to imagine the picture. A few new parts light up, but there is much in common with the pattern for seeing the actual picture. In particular, parts of the visual cortex light up — that is the imagining of the picture stimulates parts of the sensory system in a similar way to actually seeing it.BOX: fetal development of the brain, cortex as folded skin, the sea inside the children’s film Inside Out (Pixar) takes an interesting analogy of how the brain works, with memory & emotion
Memory and Time
If we consider memory we see the importance of all three forms of brain physicality: electrical, connectivity and chemical [[KV09]]:
- short-term memory – Our fleeting thoughts are carried electrically, as ionic discharges across synapses transfer information from neuron to neuron, as is measured through EEG monitoring. This is the basis of all our immediate perception, and also short-term memory with its 7+/-2 chunks of information [[Mi56]].
- long-term memory – In contrast, long-term memory is normally assumed to be stored in the physical connections between neurons and the strength of these connections. This storage literally grows, even the strength is determined by the size of the synapse.
- mezzanine memory – in between the short-term memory and long-term memory there are things between: the general sense of where I am and what I am doing and the memories of what I have been doing over recent minutes and hours that have not had time to ‘grow’ long-term memory and yet last longer than the effervescence of short-term electrical activity. This is less well studied in the psychological literature except for work in situation awareness [[En95]] and ‘long term working memory’ in studies of reading comprehension [[EK95]], indeed there is no name for this medium term memory, and the name ‘mezzanine memory’ is Alan’s own. It seems likely that this form of memory is stored using chemical build-up with neurons, a process known as ‘long term potentiation’ [[CB06,Lo03]]
Chemicals in the form of neurotransmitters are also constantly produced in our brains, and hormones flood through our bodies. Some of these chemical processes are quite fast, in the order of seconds, but some, especially those in our bloodstream, may take many minutes to return to their normal state – think how long you feel ‘jumpy’ after a scare as adrenaline and other chemicals flood through your body and do not disappear equally as fast once you realise there is no danger.
While not part of our explicit memory systems, these chemicals act as a slowly decaying trace of past feelings and activity. In particular, many of these relatively slow-acting chemicals are critical to mood and emotion, and this has its impact on digital design.
Imagine visiting a friend who is about to have a party, then moving on to another who has lost a parent. We adapt and react differently to each, but in between our bodies have had time to ‘reset’ the chemicals that influence mood.
Now imagine the same scenario, but using instant messaging chat windows. On one side of the screen planning a party on the other a funeral. While the electrical parts of our brains may be able to rapidly adapt between situations, the chemical parts cannot.
Because long-term memory is stored physically in the connections between neurons it is possible to calculate just how much a single brain can remember [[Dx05]]. We have approximately 10 billion neurons and each neuron connects to between 1000 and 10,000 other neurons giving around 100 trillion connections. It is this pattern of connections that is normally assumed to encode our memories. Each of these neurons also has a strength of connectivity given by the size of the synapse.
Now imagine a brain scanner in the far-off future that can trace each neuron including how strong they are. As it reads the neurons it gives each a number (needing about 34 bits) and records the connections and strength using a scale of 0–63, together this needs 5 bytes per connection. So with 100 trillion connections, our computer of the future would need 500 trillion bytes, or half a peta-byte, or approximately 500 hard disks. Oddly when the internet archive was set up [hilite]a few years ago[/hilite], the dump of the web was about the same size.
You can do a similar calculation for the speed of processing, to see how fast a computer would need to be to simulate the brain running in real time. The figure turns out to be about 10 peta flops, that is 10 thousand million million calculations per second. Sounds a lot, but, yes you guessed, if you add up the processing power of all the PCs connected to the Internet, this is about the same.
So in storage capacity and processing power the Internet is roughly similar to a single human brain!
Although the raw numbers are roughly similar, the web is not really like a brain (and the brain is not like the web). Our brains are relatively slow (around 100 ‘cycles’ per second), but very highly parallel – lots of things happen at once. Each PC works much faster several billion cycles per second, but serially, one thing at a time (although a 100 million PCs on the web is quite a lot of parallelism in itself!). Furthermore the brain has a lot of non-local connections, that is neurons in one part of the brain talk to ones in distant parts. This would make it very hard to actually emulate a brain on all those PCs as the Internet in between would be both too slow and buckle under the load.
Does the idea of simulating a brain worry you? It is possible to simulate fluid flows, so one could model a glass of water, but running the computer simulation of the glass of water would not make you less thirsty any more than looking at painting of a mountain stream.
The difference between simulating water and simulating our brains is that some would say that our thoughts, minds, consciousness, indeed who we are, reduces to the memory patterns and ‘execution’ of the brain functions. That is a computer emulating your brain is ‘just the same’ as you thinking it in your head.
Of course, these are things hotly contested in the philosophy of consciousness, with some arguing that the special nature of neurons makes a difference [[Se97]], or even, according to Roger Penrose, that it is the quantum effects of sub-cellular microtubules that allow us to think in ways that are not computable and therefore not able to emulated by a computer (except perhaps in a quantum computer) [[Pe90]].
When we discuss “mind and body” we shall see a different view that sets the human body as a crucial part of the picture, and some argue that no emulation of the mind could be truly a mind without being connected to a living, acting body.
We will discus the particular ways we think about objects and about space later in this book, but it is interesting to note that there are specific ways in which we think about objects that are not the same as the way we think about space, and different again from the way we think about other people. We pick up objects, move them around, and expect that they may be moved by others or natural forces. In contrast, for space, buildings, the topography of hills and mountains, we are thinking about moving ourselves around and within them.
To some extent these can be seen as workings of the same sort of reasoning applied to different kinds of situation and indeed this is certainly the case for more logical thinking. However, there is strong evidence, that we also have quite specialised ways of thinking about different aspects of the world, sometimes called ‘multiple intelligences’ [[Ga83,Ga93]]. Some evidence from this comes from brain science, showing there are specialised areas of the brain related to particular aspects of perceiving thinking and acting (to the extent that these are separate!), and some evidence from experimental studies which shows that effectively identical problems are solved differently if presented in ways which appeal to our different ‘intelligences’.
Imagine there is a pack of cards each has a letter on one side and a number on the other. Four cards are selected and placed on the table like this:
You now need to verify whether the following rule is true of the four cards:
if there is a vowel on one side then there is an even number on the other
What cards do you need to turn over to verify the rule?
This puzzle is called the Wason card test [[Wa66]] and variants of it have been used heavily in psychological experiments.
In fact, about three quarters of people get the answer wrong. The most common answer given is the ‘E’ and ‘2’, but the ‘correct’ answer is actually the ‘E’ and ‘7’. The reason for the ‘incorrect’ answers is because we have tendency to look for ‘confirming’ evidence, so we look at the back of the ‘2’, even though if it were not a vowel it would not invalidate the rule. Also we focus on the cards most similar to concepts mentioned in the question namely the vowel and the even number. Actually in the real world, full of irrelevant information, these are sensible heuristics, but they do not work so well for artificial logic problems like this one.
Now consider a different problem. A newspaper stand has an ‘honesty box’ to put in your money when you take a newspaper. A social psychologist watches people walking by. For each person the researcher writes on a small card. On one side she writes whether the person paid any money, on the other whether they took a newspaper. She even records a card for people who simply walk past.
Four of these cards are placed on the table:
You now need to verify whether the following rule is true of the four cards:
if the person takes a newspaper then they have paid money
Just as in the Wason card test your task is to turn over as few cards as possible to verify the rule. Which cards do you turn over?
The problem is formally the same as the normal Wason card test, just substitute ‘paper’ for vowel, money for even number. However, this time about three quarters of people get the right answer. The explanation given for this is that this variant recruits knowledge about fairness and cheating and so is able to tap into our social intelligence rather than abstract logic.
Some ‘evolutionary psychologists’ talk about our brains being not so much a general-purpose tool (like a survivalist’s machete), but more like a Swiss Army knife, with specialist tools to tackle special tasks [[Co89, TC97]].
Is the brain like a Swiss army knife – special part for special purposes?
In “The Prehistory of the Mind” [[Mi96]] Mithen argues that one of the crucial human developments has been the integration of these different intelligences, which in other animals including early hominids were distinct and unable to ‘communicate’ with one another. In other words the defining human cognitive accomplishment is joined up thinking!
Mithen’s analysis is not merely speculative, but based on detailed analysis of the palaeontological record. Early hominids over a million years ago were able to shape flint tools showing advanced forethought. Indeed even chimpanzees do this in the wild, selecting sticks of an appropriate size and shape, stripping them of leaves and breaking them to length so that they can later be used to dig out termites. Tool making itself is complex, but not uniquely human. However, it was not until around 60-70,000 years ago that modern homo sapiens tied flint heads to wooden handles to make better axes or spears. This is seen as evidence that wooden sticks (previously living things) could now be thought about together with physical (non-living) stones.
Similarly it is only recently that we see the making of clay or stone objects that represent animals or people and the emergence of cave art. The representing of an animate thing (creature or human) using an inanimate medium (clay, stone or paint) again shows evidence of joined-up thinking.
To describe the cognitive architecture of the mind Mithen uses a metaphor of a cathedral with (in order of evolutionary development) a nave of ‘general intelligence’, several chapels for the specialised intelligences, and a ‘super chapel’ of meta-representation where the differing intelligences can intermingle with each other leading to ‘cognitive fluidity’. The form of this meta-representation is not elaborated although Mithen suggests it is in some way related to language, and this accords with the social changes occurring at around the same period 50-60,000 BC which are again often attributed to language development, a sort of socio-linguistic Eden.
Indeed the most obvious meeting point of different kinds of intelligence is in rational logical thought mediated by language. Words by their nature make things equal; whilst very different in meaning, “stone”, “spider”, “speed”, “society”, and “science” are all ‘just’ words: tokens that can be put inside similar constructions. We can reason equally with all and can mix them together leading to a level of integration across intelligences [[fg:logic-as-focus]].
[[@fg:logic-as-focus]] Logic as the focus
Given the topic of this book is physicality, as you can imagine that is not the end of the story.
Simpler creatures than ourselves, such as dogs, birds, and fish, do not lead fragmented existences, but are integrated because they live in the world. When we think based on one sort of understanding, then the results of the actions are available to all of us. If you want to go to see a colleague (social intelligence) and they are in the next room, you may start to walk towards them, but the fact that a wall is in the way will immediately be obvious to your spatial understanding. …and if you do walk forward the actual presence of the wall will make it even more obvious. Things do not need to be linked in the head because they are linked in the world (fig. [[fg:world-kicks-back]]).
[[@fg:world-kicks-back]] The world kicks back
However, whilst we do not need to think in our heads, of course we often do, but apparently without lots of intense logical thinking. Partly this is due to imagination. As soon as you consider going to see your colleague next door an image starts to form of going. However, recall that imagination works its way right back into the sensory system, so it is almost as if you are actually seeing it, and this is available to your spatial intelligence just as with the real world. The physical intelligence says “no” without you ever getting up from your chair. The ‘virtual world’ in your head kicks back just as hard as the real world (fig. [[fg:imagination]]).
the brain as interface
Most of our interactions with computers use our limbs or maybe our voice to input to control the computer and sight, hearing or occasionally touch to get feedback from the computer. However, there are a number of technologies that more directly connect computer to brain, by passing our voluntary body movements and our normal senses.
We have already seen one example of this, the implants directly into Kevin Warwick’s arm nerves that enabled him to control devices and directly connect to his wife. The nerves here were ones that were normally used to control muscle movement and bodily feelings. So in a way this is bodily interaction … just bypassing the actual body!
With different technology one can even bypass the nerves and tap directly into the brain, this is often called “brain–computer interaction”.
We discussed how different parts of the brain perform different functions. Brain scanners can reveal this, but are a little large for use in computer interfaces, however, simpler EEG equipment can be used. EEG (electroencephalography) is used for medical diagnosis and involves placing large numbers of electrodes on the skull and measuring the electrical activity induced in the scalp by the corresponding activity within the brain due to the firing of neurons. Clinical apparatus may involve many dozens of electrodes, but for computer interaction research, useful information can also be obtained using just a small number, often fitted into a skull-cap.
For many years this has been used in simple bio-feedback equipment, perhaps playing a different tone or projecting colours. In particular this is often used together with the detection of particular frequencies of activity (alpha and beta waves) associated with states of alertness or relaxation. While we do not normally have conscious control over these states of the brain, once they are made in some way visible or audible, subjects can often learn to influence them.
Bio-feedback is more about allowing the user to monitor their own body. However, if the outputs of EEG equipment are put into a machine learning algorithm, it is possible for the computer to recognise some of the users ‘thoughts’ … but only from a small pre-chosen set. For example, using apparatus of this kind there has also been some success in controlling an on-screen pointer by having subjects simply think about it moving one way or another. The recognition rate is still relatively poor, so this is not currently an option for day-to-day computing. For those patients with severe disabilities, particularly those with ‘locked-in syndrome’, both non-invasive use of EEG and similar apparatus and also more direct brain implants offer a chance to be able to communicate with the world.
For the blind there has also been some success. Visual images are transferred from the retina to some parts of the visual cortex in a fairly direct way: that is there is something like (but not exactly like) a little image of the scene that you can see in front of you in the visual cortex. By working the other way round, brain implants that stimulate the visual cortex appear as points of light.
Less invasive means can be used for more day-to-day interactions, especially in affective gaming, that is games that respond in some way to the user’s emotional state. Most of these make use of heart beat, or GSR (galvanic skin response — basically sweating as used in lie detectors). Both are related to the general level of arousal of the body, and so can be used as a rough measure of emotion. The most common form of game is ‘relax to win’, as you become more relaxed you get more speed, or energy. This may be combined with social situations in which your friends try to make you less relaxed!
These physiological measures tell you how strong an emotion is, but not whether the person is very happy, or very angry, and certainly not subtle emotions such as apprehension. Typically more detailed emotion detection needs to use facial expression or tone of voice, both more complex, and still only delivering a fairly small palette of emotions. Kiel Gilleade’s research has tried to bridge this gap by using the fact that as humans we are successful (when we are!) at detecting other people’s emotions, because we also know the context [[GDA05]]. In games one has a lot of information about the context as it is a simulated world, and hence it is possible to make some inferences even from simple physiological measures.
The trouble with attempting to ascertain brain state from body state is that the body is affected by many things as well as what you are thinking and feeling. A researcher at Bristol University, Cliff Randell, designed a small camera that he wore connected to a heart rate monitor. He reasoned that the important events one might want to record would be marked by periods of heightened emotion, so a camera was triggered to take a photograph when his heart rate peaked. The technology worked in the lab, but when he wore it in real life he found the photos were full of the tops of flights of stairs.
While the lacuna between pure mind and computation remains, it is perhaps not as wide as it once was. The means of contact are still very physical (silicon chips in the brain, wires and electrodes on the skull), but intimate and direct.
creativity and physicality
Creativity is surely a defining cerebral activity. Artists wait upon the inspiration of their muse and the results are translated into words, images or sound. But of course it is not quite like that. Painters rarely sit and then paint, but just paint, they seem to think with their brush. Similarly a composer will write their score sitting at a piano and a poet will jot down phrases. We will discuss this embodiment of thought more generally in the next section, but certainly there is some connection between creativity and physical manifestations.
The same is true of problem solving. Some problems can be solved analytically, by decomposing the elements of the problem and then going fairly mechanically through cases. However, others are not amenable to procedural solutions. Psychologists talk of ‘insight problem solving’, puzzles that seem intractable, but where there is a sudden “ahah!” moment and they become instead obvious. Look at the matches below. If you remove the seven matches from the top boxes, it is easy to leave just three squares left. But can you remove five matches to leave exactly three squares?
You may try and ‘do it in your head’, but it is more likely that you will experiment and play with real matches. You might need some time away from the problem too, what the psychologists call ‘incubation’ and then return to the problem and it becomes easy. So both time engaged in the physical form of the problem and time completely divorced from it seem to be necessary.
One such psychologist, Ronald Finke, studied this very physical ‘playing with materials’ stage in more creative design [[Fi90]]. In a series of experiments he studied simple problems where subjects were asked to perform simple creative tasks: creating things (e.g. means of transport) from shapes, such rods, circles, etc. In some conditions the subjects were first given the shapes to play with for a period and then asked to come up with a design. In others they were simply presented with the shapes and straight away asked to come up with a design. He found that those who had time to play in a non-goal-directed way were able to produce more ‘creative’ designs.
This series of experiments by Finke and his colleagues, were the basis of the ‘geneplore’ model of creativity [[FW96]], which has two stages. The first is a ‘generation’ phase (corresponding to the play), which is not directed towards the goal, during which the subjects develop ‘pre-inventive structures’ patterns and relations of the materials to be used. This is followed by an ‘exploration’ stage (slightly confusingly named) when the pre-inventive structures are put together to deliver the creative solution. While there are criticisms of both the model and the methods, the work underlined the benefits of the use of physical materials (although interestingly people could also performed purely mental ‘play’ as well).
In our own observations of group design activities we have also seen profound effects of the choice of design materials and the results obtained. As part of a workshop on the topic of Physicality, attendees were divided into small teams and given a design brief “a hand-held device for producing light”. Each team was given one of three kits of design materials to use, either (a) paper and pencils, (b) card and glue, or (c) modelling clay (plasticine). Participants were only supposed to use their own materials (although some cheated!). Beyond this they were not told how to use the materials, but in fact the materials implicitly suggested ways of use – for example, no team in the paper and pencil group chose to fold or mould the paper to make a model. In normal design any or all of these materials would be used according to the preferences of the designer at a particular moment. However, in this exercise, teams of participants were given just one kind of material to work with. Thus we were performing something similar to the ‘breaching experiments’ [[Ga67]] discussed in the Introduction: a deliberate disruption of human activity in order to bring to light aspects that are tacit or taken for granted. In this case we were disrupting the ability to choose appropriate materials.
Analysis of this design exercise and other work reveals the subtle ways physical materials influence design [[RH08]], in particular, we saw differences in the way groups investigated depth vs. breadth of ideas (paper and pencil sketches and lists generated more ideas than physical materials, but the latter were elaborated more). There were also subtle ways in which the materials both constrained but also suggested ideas (rolling of paper, deformability of clay), reflecting the importance of constraints seen in the creativity and problem-solving literature [[OM02]].
Note we carefully use the word ‘influence’, as the materials do not determine outcomes; for example, one group given physical materials ‘cheated’ and used paper and pencil to sketch, reflecting Buxton’s focus on the importance in early design of the indeterminacy of the sketch [[Bx07]] over the more concrete prototype. And human ingenuity can conquer the obstinacy of the material world, one group made a cuddly teddy bear out of paper by tailoring with the paper as one would a suit of clothes.
Solution to match puzzle in Figure [[fg:match-puzzle]]]
[[Bx07]] B. Buxton. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann, 2007
[[CB06]] S. F. Cooke, T. V. P. Bliss; Plasticity in the human central nervous system, Brain, Volume 129, Issue 7, 1 July 2006, Pages 1659–1673, https://doi.org/10.1093/brain/awl082
[[Co89]] Cosmides, L. (1989). The Logic of Social Exchange: Has natural selection shaped how humans reason? Studies with the Wason selection task. Cognition, 31:187–276
[[Dx05]] A. Dix (2005). the brain and the web – a quick backup in case of accidents. Interfaces, 65, pp. 6-7. Winter 2005. http://www.hcibook.com/alan/papers/brain-and-web-2005/
[[En95]] Endsley, M. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32-64.
[[EK95]] Ericsson, K., and Kintsch, W. 1995, Long-term working memory. Psychological Review, 102, 211-245.
[[Fi90]] Finke, R.. Creative Imagery: Discoveries and Inventions in Visualization by # Lawrence Erlbaum (1990). ISBN-13: 9780805807721
[[FW96]] Finke, R.; Ward, T.B. & Smith, S.M. (1996). Creative cognition: Theory, research, and applications. MIT Press. ISBN-13:978-0-262-56096-2
[[Ga93]] H. Gardner. Multiple Intelligences: The Theory in Practice. NY: Basic Books. 1993.
[[Ga67]] Garfinkel, H. (1967). Studies in Ethnomethodology. Englewood Cliffs, NJ: Prentice- Hall.
[[Ga83]] H. Gardner. Frames of Mind: The Theory of Multiple Intelligences. New York: Basic Books. 1983.
[[GDA05]] Gilleade K, Dix A, Allanson J (2005) Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me. In: Online Proceedings of DIGRA 2005, June 2005
[[KV09]] Katifori, A., C. Vassilakis and A. Dix (2009). Ontologies and the Brain: Using Spreading Activation through Ontologies to Support Personal Interaction. Cognitive Systems Research (in press). http://www.hcibook.com/alan/papers/Ontologies-and-the-Brain-2009/
[[Lo03]] T. Lømo, The discovery of long-term potentiation. Philosophical Transactions: Biological Sciences. 358(1432) (2003) 617–620.
[[Mi96]] Mithin, S. The Prehistory of the Mind. Thames and Hudson, 1996.
[[Mi56]] Miller, G. The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. The Psychological Review, 1956, vol. 63, pp. 81-97. http://www.musanim.com/miller1956/
[[OM02]] Ormerod, T., MacGregor, J., and Chronicle, E. (2002). Dynamics and constraints in insight problem solving. Journal of Experimental Psychology: Learning, Memory, and Cognition, 28(4), 791-799.
[[Pe90]] Roger Penrose The Emperor’s New Mind, Oxford University Press, 1990
[[RH08]] Ramduny-Ellis, D., Hare, J. Dix, A. and Gill, S. (2008) Exploring Physicality in the Design Process. In Undisciplined! Proceedings of the 2008 Design Research Society International Conference, (16-19 Jul. 2008), Sheffield, UK.
[[Se97]] J. Searle, The Mystery of Consciousness, Granta Books, 1997.
[[TC97]] Tooby, J. and L. Cosmides. Evolutionary Psychology: a primer. 1997. on-line at: http://www.psych.ucsb.edu/research/cep/primer.html
[[Wa66]] Wason, P. C. (1966). Reasoning. In B. Foss (Ed.), New horizons in psychology. Harmondsworth: Penguin Books. pp. 135–151
use journal paper instead for [[RH08]]