wearable | handheld | intimate | secret | personal | expectant | response | system
wireless | heuristic | invisible | sensory | private | environmental | reproducing | system

networked | datamining | machine learning | archiving |
remote sensing | telepathy | physical response| invisible | desire
installation | performance| mapping | unearthing | dynamic database

Abstract

This essay explores the conceptual, technological and physical processes of the whisper-project. whisper is a major collaborative art research project based on engineering small wearable devices and handheld technologies resulting in a participatory installation.

whisper is a [wearable, handheld, intimate, sensory, personal, expressive, responsive] system which constructs networked messages based on inferred states of carrier bodies - the hosts for small wearable devices. whispers are wearable body architectures.This essay elucidates the creative and collaborative processes which include collective first person methodologies, and our version of the 'sewing circle'; the phenomenon of the participatory installation as an emergent non-hierarchical performative form; the aesthetic of whispers which emerge directly from handworked fabrication of the materials: (sensors, circuits, electronics embedded in latex, silicon, rubber, and the body) in a play across the opaque, translucent, transparent; and reconfiguration of attitudes toward the body which allow for our corporeal selves to be seen as fluid, networked and dynamic systems with concealed information to be unearthed and mapped onto linked and networked devices.whisper appropriates the attention, breath, brainwaves, heartbeat and affective qualities of a community of participants, rewriting them as shared signals on the network, archived and reconstructed. Data flows are generated, represented and archived in intimate connection with the bodies that produce and alter them.

The concept of 'pulp fashion' is a metaphor for the impermanence of physical states. We borrow from Maurice Merleau-Ponty's concept of the invisible and pulp of the sensible. whisper 'pulps the sensible', defrocking the habitual, shredding our non-physical rationality and 'nips and tucks' at the subtleties of our bodies. As devices are designed to link our breath and gesture, pulp fashion also becomes a source of 'inspiration'. whisper devices intimate the invisible as tactile and kinesthetic. Flesh is the connective tissue spanning the organic and the inorganic.

Introduction

"Visible and mobile, my body is a thing among things; it is caught in the fabric of the world, and its cohesion is that of a thing. But because it moves itself and sees, it holds things in a circle around itself. Things are an annex or prolongation of itself; they are encrusted into its flesh; they are part of its full definition; the world is made of the same stuff as the body". (Maurice Merleau-Ponty, 1961:163)

whisper is a participatory installation based on small wearable devices and handheld technologies. whisper[s] are wearable body architectures. whisper constructs networked messages based on inferred states of carrier bodies - the hosts for small wearable devices. whisper takes place in an installation space and on the web. It generates and represents data flows in intimate connection with the bodies that produce and alter them, building a dynamic persistant data structure, a living learning archive of body and system state. whisper builds upon physical practices such as dance improvisation, and manifests cultural and scientific theories of embodiment.

In its research + development phase, whisper aims to unearth physical data patterns of the body, mapping that data onto linked and networked devices worn close to the surface of the skin. Collections of wearable devices will be networked together, between bodies, or traversing a single body. A range of physiologically based input signals will be explored: breath, pulse, brainwave patterns, electrical energy, and temperature. whisper appropriates the attention, breath, and heartbeat of a community of participants, rewriting them as shared signals on the network. whisper extrapolates from the body onto the larger collection of networked bodies: effectively a performance piece in a social space.

Both the input and output components are worn close to the skin, on clothing, around the neck, wrist, or ankles, like jewelry, attached to a piercing, next to the heart, or on one's sleeve. Through the use of small motors and sensors, the components can output vibrations, temperature changes, sound, light, color, miniature texts & images, even low-bandwidth video displays. Basic analogue devices will be used alongside more sophisticated components (including biofeedback - or brainwave - sensors). A wearer may configure their plugout device(s) to vibrate, tickle, or sigh when it receives data associated with a particular pattern set. Maximum configurability is accomplished by 'plugging in' components, by mixing and matching functions within this modular system.

whisper plays in an ironic way with domains of influence, visibility, and the shifting threshold between the tangible and intangible through the 'carriers' threshold of perception. whisper shifts our attention to this mobile threshold and introduces concepts of future memory as a function of mapping previous states and extrapolating them into present interactions. Future memory is represented within a dynamic living semantic data archive. As the whisper system evolves over time, it will be able to re-visit its past decisions, in the light of current intentions, and recover past potentials obscured by the initial instantiation of behavior at that time. Not everything can be known at the point of its initial enactment; the past is incomplete and the whispers can revisit and reconstruct past views as it progresses. The past is not replaced, it is augmented and restructured as the system perception grows. And the rediscovery of the past propagates into the future and the system's anticipated behaviors. Intention is constructed, communicated and functions to direct and apply whispered messages which range between direct and subliminal, suggestive and overt, seductive and definitive. Emerging behaviors are created and based on sharing sense-based communications between bodies that emanate their softly voiced messages within a space. The distributed network shares layers of messages within the future memory: deconstructing the notion of archive to 'shapeshift' temporal representation to include a representation of future (and past) as a function of present.

The overall aesthetic of whisper emerges through work with materials. Explicitly stated: The aesthetic of this project is generated by process and practice. We deliberately avoid conventional wearable aesthetics, such as cyberpunk or the current sports accessory look. Our aesthetic comes from designing for invisible connections. It is emergent and ambiguous, almost a 'reconstructed feminine', but with hard edges; it is a juxtaposition of what seems like the 'soft organic' with the 'soft & hard inorganic'. The look and feel emerges from our materials: latex, silicon, rubber, paper, circuits, wires and exposed sensors. Colours include: amber, white, clear, milky, some black and pink. The transparent, translucent and the opaque converge with skin.

"I cannot deny a taste for the overlaid and the incised: translucency as well as total transparency". (Hawley, 1998:169).

Skin is the richest source of inspiration: marked, mapped, extended, exposed, nurtured and celebrated. Ultimately, our devices will be fetish items, things people will desire to take home with them to relive their experience of whisper.

"My flesh and that of the world therefore involve clear zones, clearings, about which pivot their opaque zones, and the primary visibility". (Merleau-Ponty, 1964:148)

Research & devising [fabricating] process

The research+development fabrication process for whisper is based around the simultaneous creation of hardware, software, movement, and textural, material vocabularies. This process simultaneously and consciously refines and develops iteratively in relationship to itself. This reflexive turn is fostered through physical acts, motivations, fabrications and processes. We are committed to social design, performance design, and technical design through materializing the whisper project. We craft collective first person methodologies as processes and strategies for collaboration across scientists, technologists and artists.

The 'first person' of these methodologies comes into play through emphasis on design that is intimately connected to the body. Like phenomenology, collective first person methodologies are based primarily upon physical experience, but emphasis is shifted to the collective rather than the individual unit. Each stage of the research period is linked to exploration in the (movement + electronic) studio. Physical improvisation techniques determine emergent movement vocabularies and inform the design process. The process is not simply to import pre-fabricated devices into the studio. All materials and devices are tested physically so that the body knowledge and the hardware/software design occur simultaneously. The creative development of the wearable devices is an embodied and performative process.

We call our process the sewing circle. Generally attributed to groups of women, domesticity and textiles, the term is associated with 19th century social and creative processes. This term is employed in the interests of rehabilitating a largely dismissed creative activity. The implied message is of shaping and building an artifact according to an inherently social and collective design process. Like the members of sewing circles and other creative collectives, we are building our own vocabularies, physical techniques and methodologies. We are also committed to working with textiles and mapping the skills of knitting and stitching onto device design. Our sewing circle may stitch latex and knit with rubber, but we will also wire our bodies into wearable devices and physically improvise fabricate and engineer in the studio.

"The effort to create apertures became much more deliberate, and the pathways of both heat and light emerged as an essential component rather than as the result of applied aesthetics." (Hawley, 1998:173)

The whisper design process iterates across physicalisation, conceptualisation, device construction, and software development. These are no longer first generation wearables which were basically 'one-liners' (strap them on and get the point immediately). The whisper devices awaken in us the knowledge that we dwell in/with/through our bodies, and with dwelling comes a commitment to building relationships. Learning any physical technique is a process of building relationships between our centres of gravity, visual perception, kinaesthetic perception, muscle tension, and communication with others. We become liminal beings as our layers of subtle body knowledge reconfigure with any new technique, orchestrating the internal with the external until these distinctions dissolve.

"movement that touches and movement that is touched". (Merleau-Ponty, 1964:148)

Pulp fashion

The concept of 'pulp fashion' is a metaphor for the impermanence of physical states, and as such, is for us vastly innovative and seductive. To fashion is to engineer, create, to sew, to ergonomically render, through a process of nips, tucks, slices and stitches. Enough has been written about commericalization and commodification of the fashion object (and more can always be heard about fettishization). The opulent is fundamentally not about cost but about sensuality. Fashion is the painstaking crafting of the changeable, or the ephemeral. Our bodies are ever changing fields: chemico-perceptual shifts, mood swings, neuro-physical peaks and valleys, kineto-tactile data feeds. Stasis is a myth. We can hardly keep up with ourselves. The suggestion that we are a nexus of information flows is accurate, but only partial. We are these, but we are flesh at the same time. Flesh is not the 'meat' of cyber-lore. Flesh slips through and smothers the tired dualities of mind/body, consciousness/meat. We exist in a network of flesh.

Maurice Merleau-Ponty's enigmatic understanding of flesh encompasses bodies, organic and inorganic objects and what used to be known as space.

"... the flesh is not matter, is not mind, is not substance. To designate it, we should need the old term 'element,' in the sense it was used to speak of water, air, earth, and fire ...". (Merleau-Ponty, 1964:148)

Flesh contains within its very fabric the connection we feel with the environment, the perceptual play of light and shadow, our embodied selves and the bodies of others. It is impossible to understand our flesh as it enters into an exchange with the whisper devices without the related phenomena of the invisible and the hidden. The goal, when accessing the invisible (or 'subtle') is not simply to bring it to the surface so that it becomes visible, just as the point of whisper is not to de-frock the subtle but rather to work in an ambiguous territory, within the realm of the physically and semantically tentative. This approaches the state between waking and sleeping, allowing oneself to remain in that state to witness one's own participation in an alternate flow of activity. A rendering 'literal' of deeper body knowledge would be a simple translation into language and would strip it of its essence. The whispering between devices based around the pluggable/unpluggable and reconfigurable inputs and outputs, give us the choice of acknowledging the invisible while retaining its hidden, or subtle, qualities. Merleau-Ponty again gives shape to the subtle through his consideration of the invisible.

"the invisible is

1. what is not actually visible, but could be (hidden or inactual aspects of the thing - hidden things, situated 'elsewhere' - 'Here' and 'elsewhere')
2. what, relative to the visible, could nevertheless not be seen as a thing (the existentials of the visible, its dimensions, its non-figurative framework)
3. what exists only as tactile or kinaesthetically, etc ...".

Participatory installation

The space of the installation can best be described as a networked ecosystem containing input and output devices. Up to 12 participants may move and browse in and out of the space with access to a set of devices which may sewn into a jacket, cape, hang from the grid above, or may be lying on the floor waiting to be picked up. Sound is generated on the body and is affected by the networked data as it begins to be transmitted. The devices will resemble a cross between cyber-jewellery, exquisite art objects, creepy prosthetics, peculiarly ornate theatrical costumes and body sculpture; they will be a weave of analog and digital components and circuitry. Participants visiting the space enter a community of bodies and objects, which have a functionality that is not yet manifest. They will be invited to take their place within this ecosystem.

Instead of situating the participants within the flow of a pre-scripted event, they will be involved in a conceptual, physical, aural and visual journey that unfolds according to their participation as a body, as a system. Their responses will drive the experience, and encourage the development of other senses within our synaesthetic matrix of sensory perception and proprioception. It is for this reason that we have coined the term 'participatory installation' to describe form of whisper. Merleau-Ponty has described "a being by porosity" wherein our bodies and the distances between ourselves and others "participate in one and the same corporeity or visibility in general" (Merleau-Ponty, 1964:148).

Emanating relationships

Interaction in whisper is based on creating and emanating relationships through subtle exchanges between the small devices. Each whisper device emanates their state to the other whisper devices that are within range. The receiving whisper perceives and incorporates the state space of the other(s). The state of the whisper device is a direct function of the body that wears it, along with the memory and future memory. Emanation is abstract but perceptible, it implies a sender (source; originator) and a receiver (of the abstract and perceptible thing), and suggests outgoing and incoming signals.

Each device has a whisper device-state, and the collection of devices defines the current global whispers system-state. Device states emanate from or are whispered to other devices within proximity of the range of influence. whisper devices also 'perceive' these emanations based on their current state, and alter their own state, based on incoming perceptions. whisper device states are learned and emerge from living on a specific body, and begin to represent that body. The whisper devices also remember past bodies and states, and these past lives influence their behavior. These states form the basis for the persistent database archive.

The device is a gadget, tool, apparatus, mechanism, appliance, gear, invention, analytic tool, metaphor and literary technique; the device is gendered, an extension of the self, an interface, prosthetic and fetish object. It is an object of industrial design. The design of the device incorporates dominant cultural narrative within its functional and aesthetic properties including the displacement of taboo and desire; the device is a network object, a browser, a search engine, a system and theatre, the device is thought, is telepathy, is our own body.

Our body as system creates a metaphor for the operating model of whisper. Our bodies are composed of multiple networked systems, which communicate autonomically to and with each other. So do the devices in whisper. Our bodies are shaped with multiple thresholds that operate in stealth at one moment, overtly at the next moment. These thresholds lie at the liminal boundary of our perception. Our bodies are fluid, networked, and dynamic. Our bodies have secrets, contain multiple intelligences, conceal information in unlikely places, and develop strategies for the expression of current and archived states. So do the devices of whisper. Our bodies surrender things to one another. Our bodies learn, habituate and unlearn by applying directed attention. So does whisper. Any one of our bodies is a 'we'. When our bodies are together they can operate as an 'I'. So can the devices in whisper.

The whisper system consists of a group of interconnected, autonomous wearable sensor | actuator collections combined with a local processing module which gathers the data from multiple sensors and whispers to it's neighbors, and a central whisper server, via low-power radio frequency (RF) transceivers. The server, database and micro-controller systems together determine the response to the actuators. They provide feedback in the form of sound, graphics and tactile feedback devices that are worn on the body, along with the sensors. The whisper central server also visualizes and displays a dynamic representation of the system-state both on the web, and projected onto a series of heterogeneous projections displayed within the installation space.

Future memory

In whisper, the concept of future memory is represented through the dynamic visualization of its own system state within a living semantic data archive. whisper maintains and displays a dynamic representation of its memories, the computational equivalent of precognition, electronically augmented telepathy. The whisper devices and participants, through their behaviors anticipate potential future behaviors and state. The state flows from the past to the future; the intentions are always dynamic, as velocities and accelerations of behavior. The distributed network shares layers of messages within future memory: deconstructing the notion of archive to 'shapeshift' temporal representation. This includes a representation of future (and past) as a function of present.

As the whisper system evolves over time, it will be able to re-visit its past decisions, in the light of current intentions, and recover past potentials obscured by the initial instantiation of behavior at that time. Not everything can be known at the point of its initial enactment; the past is incomplete and the whispers can revisit and reconstruct past views as it progresses.

The past is not replaced, it is augmented and restructured as the system perception grows. And the rediscovery of the past propagates into the future and the system's anticipated behaviors.

In whisper, communication is characterized by its context: whispers can be qualified (in contrast to quantified). Qualified communication deals with intent: gesture, tone, pitch, repetition, redundancies: are all qualified elements of communication because they provide context.

Unearthing | anarchiving

whisper is a foray into the cultural study of telepathy and of mapping techniques: impressions are transferred invisibly, mediated both through body and technology. The research process explores invisible datastreams as wireless networks and suggests, both playfully and literally, that telepathy is the ultimate wireless network. whisper builds wearables for the telepathically impaired: as 'aware-able' devices, they make bring to awareness functions of embodiment and perception that were previously ignored.

The concept of mapping as a dynamic unearthing knowledge is critical to the design of this research and is related to the 'aware-able' device. Mapping is a technique that can help to reveal or define underlying patterns of processes and information. Mapping offers a new view on an idea, a process, an event, an object or a place. Maps provide a means of visualization that might unearth patterns within one of these views. Mapping is a discovery: it may reveal new knowledge within an area thought previously to be known, or it may help in the acquisition of knowledge or experience of what is not known. Rediscovery within known areas is often achieved by combining views which might, at first seem irrelevant to each other, like mapping processes that are not normally regarded as important. This research will map data signals from collected and networked bodies, using sensors that collect physiological data. This reflects the awareness that our bodies are subtly evolving maps of our identities and our lives. whisper excavates the invisible, is a search for lost things.

Collaborators:

whisper is a collaborative co-production with th eRotterdam Production House: V2_ and the Rotterdamse Schouwburg (Rotterdam City Theatre). The project includes a range of skills and competencies from movement and performance and physical design, hardware and electronic design, to software engineering and programming, physical space design, data architecture design, sound design, industrial design, and mathematical visualization. The whisper team of collaborators include: Thecla Schiphorst, Susan Kozel, Andruid Kerne, Kristina Anderson, Julie Tolmie, Norm Jaffe, Diana Burgoyne, and Sang Mah. The V2_ co-production collaborators include Anne Nigten, Stock, Erik Kemperman, Stephan Drescher, and Ruben de la Rive Box.

Bibliography:

Hawley, Christine. 1998. Invisible Lines, in The Architect: Reconstructing Her Practice, ed. F. Hughes. MIT Press.

Merleau-Ponty, Maurice. 1961/1985. Eye and Mind, in The Primacy of Perception, ed. J.M. Edie. Evanston: Northwestern University Press.

Merleau-Ponty, Maurice. 1964/1987. The Visible and the Invisible, ed. C. Lefort. Evanston: Northwestern University Press.

0. The archive will speed us up - will halt us

If you try to imagine what life will be like hundred years from now, it must be frightening. For one, everyone can carry on in its pink-top an archive bigger than the life-long capacity of the senses. So, the relevant portion of the information reaching us can be stored and processed off-line. This must have the effect on the one hand that the processing of information and hence of life will proceed in an even faster pace. And, at the same time, it must have the effect that the development of life will be slowing down as all information is stored in an archive with its tendency to conserve what is there. (In 2002, on the one hand is the effort to write, rehearse and perform a theater play within 24 hours. This is the ultimate expression of the progressive archive-less times we live in. At the same time, the Dutch government requires now that all expenditures are accounted for, leading to extreme archive-conservatism.)

Archives are waiting for us as a huge giant asleep. If we wake the giant, what is it able to tell us? What language will it speak? Will we be able to understand the dreams it has in its mind? And, the giant grows and grows by interconnections. Will it be a force stronger than ourselves?

I cannot answer these questions. What I can do is give an overview of the possibilities and impossibilities of accessing a visual archive and the current bottlenecks.

1. Content based image retrieval - at the end of the early years

There is something about pictures that no words can convey. Consider Munch's The scream, or a performance by a video artist, or even the average Mondrian. It has to be seen. The same holds for of pictures of the Kalahari Desert, a dividing cell or the facial expression of an actor playing King Lear. It is beyond words. Try to imagine an editor taking in pictures without seeing them, a radiologist deciding on an X-ray from just a verbal description. Pictures have to be seen and searched for as pictures, by object, by style or purpose.

Research in content-based image retrieval today is a lively discipline, expanding in breath. As it happens during the maturation process of many a discipline, after early successes in a few applications, research is now concentrating on deeper problems, challenging the hard problems at the cross roads of the discipline from which it was born: computer vision, databases, and information retrieval.

At the current stage of content-based image retrieval research, it is interesting to look back towards the beginning and see which of the original ideas have blossomed, which haven't, and which were made obsolete by the changing landscape of computing. In February 1992, the USA-based National Science Foundation organized a workshop in Redwood, CA, to identify major research areas that should be addressed by researchers for visual information management systems that would be useful in scientific, industrial, medical, environmental, educational, entertainment, and other applications. There are earlier attempts, such as the 1979 conferences on databases and pictorial applications in Florence, but nothing of much interest for today was reported there. In hindsight, the workshop can be marked as the beginning of research in content-based retrieval.

Why did it take so long to get the exploration of visual material started? Before 1995 the machine power, the capacity and reach of the Internet and the availability of digital sensors was underdeveloped to bootstrap any serious use of image exploration. But just after the NSF-workshop, things were to change quickly. The Mosaic Internet-browser was released spawning the web revolution that very quickly changed all cards. In the same era a host of new digital vision sensors became available. The number of images that the average user could reach increased dramatically in just a few years. Instantly, indexing tools of the Web or digital archives became urgent. And, in science, the visual image databases and exploration of the visual content has drawn considerable attention ever since. In order to appreciate the current state of affairs we need to discuss some basic observations which hold true for all images and all observers - man or machine alike: the sensory gap and the semantic gap.

The sensory gap is the gap between the object in the world and the information in a numerical/verbal/categorical description derived from an image recording of that scene. A computer can only process the numerical information derived from an image, so it is important to realize how much information is lost when converting an image into a digital description. For narrow image domains with a limited and predictable variability in its appearance a special digital language might be developed, but still the challenges are considerable. In the narrow domain of frontal views of faces, they are usually recorded against a clear background and illuminated with white frontal light. Where each face is unique and has large variability in the visual details, there are obvious geometrical, physical and color-related constraints governing the domain. Still the same person could render a thousand different recorded faces depending on mood, beard, weather, time of day, make-up, glasses, lighting position, incident shadows, clothes, hair-cut, frame of photography and so on. And this is yet a relatively narrow domain. The domain would be called slightly wider had the faces been photographed from a crowd or from an outdoor scene. In that case also clutter in the scene, occlusion and a non-frontal viewpoint will have a major impact on the digital description. For a broad class of images, such as the images in a photo-stock, the gap between the feature description and the semantic interpretation is generally wider still. The sensory gap makes the description of objects essentially uncertainty in what is known about the object. The sensory gap is particularly poignant when a precise knowledge of the recording conditions is missing. The 2D-records of different 3D-objects can be identical. Without further knowledge, one has to decide that they might represent the same object. Content-based image retrieval systems may provide support in the reduction of the uncertainty through elimination among several potential explanations, much the same as has been done in natural language processing. In short, the sensory gap introduces an uncertainty in any description of an image as a thousand (slightly) different images are mapped on the same description.

I guess most of the current disappointment with standard retrieval systems originates from the semantic gap. The semantic gap is the lack of coincidence between the information extracted from visual data and the interpretation that the visual data have for a user. The semantic gap is best illustrated by pictures each holding a chair. When searching for a chair we may be satisfied with any object under that name. That is we search for man-defined equality. When we search for all one-leg chairs, we add an additional geometrical constraint to the general class. The same holds when searching for a red chair, now adding a color constraint in the search, not a geometrical condition. When we search for a chair perceptually equivalent to a given chair, at least it must be of the same geometrical and color type and we are down to the sensory gap. Finally, when we search for exactly the same image of that chair, literal equality is requested, still ignoring the variations due to noise in the image and we are in the realm of image processing. Where a linguistic description is contextual, an image in an archive is not and may live by itself. So closing the semantic gap will at least include contextual search, a topic barely touched upon in science. Systems like [Chang, Rui] are collecting images from the Internet and inserting them in a predefined taxonomy on the basis of the text surrounding them.

When sorted on their purpose of image search, we discriminate three types of systems:

1. Searches by association at the start have no specific aim other than find interesting things. It often implies iterative refinement of the search, the similarity or the examples with which the search was started. Systems in this category typically are highly interactive, where the specification may by sketch or by example images. The oldest realistic example of such a system is probably by [Kato]. The result of the search can be manipulated interactively by relevance feedback [Hiroike, Frederix]. To support the quest for relevant results, also other sources than images are employed, [Swain].
2. Target search may be for a precise copy of the image in mind, as in searching art catalogues [Qbic95]. Target search may also be for another image of the same object. This is target search by example. Target search may also be applied when the user has a specific image in mind and the target is interactively specified as similar to a group of given examples, [Cox]. These systems are suited to search in catalogues.
3. Category search aims at retrieving an arbitrary image representative of a specific class. It may be the case that the user has an example and the search is for other elements of the same class. Categories may be derived from labels or emerge from the database [Swets]. In category search, the user may have available a group of images and the search is for additional images of the same class [Ciocca]. A typical application of category search is catalogues of varieties, with a domain specific definition of similarity.

The pivotal point in content-based retrieval is that the user seeks semantic similarity, but the database can only provide similarity by data processing. This is what we called the semantic gap. At the same time, the sensory gap between the properties in an image and the properties of the object plays a limiting role in retrieving the content of the image.

We discussed applications of content-based retrieval in three broad types: target search, category search and search by association. Target search is closest to computer vision research. Category search is much more challenging and requires on-line learning or visual data mining. Search by association is hampered most by the semantic gap. As long as the gap is there, use of content-based retrieval for browsing will not be within the grasp of the general public as humans are accustomed to rely on the immediate semantic imprint the moment they see an image. New ways of presenting and learning from archives are necessary here. In general, I would formulate as the challenge for image search engines: to tailor the engine to the narrow domain the user has in mind, via query specification, via learning from past, and via current interaction.

2. The state of the art - a few practical tips and a disappointing conclusion

To enhance the image information, retrieval has set the spotlights on color, as color has a high discriminatory power among objects in a scene, much higher than gray levels. The purpose of most image color processing is to reduce the influence of the accidental conditions of the scene and sensing, that is another definition of the sensory gap as discussed above. Progress has been made in tailored color space representation for well-described classes of variant conditions. Also, the application of geometrical description derived from scale space theory will reveal viewpoint and scene independent salient point sets thus opening the way to similarity of images on a few most informative regions or points.

In the description of the image one usually starts from assuming strong segmentation of the image. The alternative to do no segmentation at all is unattractive as one mixes aspects of all objects in the image into one soup of numerical descriptions. Strong segmentation is the precise outline in the image for each individual object. This is an incredible complex task, human can do in the second they see the picture, even if they have never seen the topic of the picture before. There is no chance a machine can perform this general task in a hundred years, simply because humans use one third of all the brainpower to achieve this. Luckily - and it took years to realize this - for retrieval a total understanding of the image is rarely needed. Of course, the deeper one goes into the semantics of the pictures, the deeper also the understanding of the picture will have to be, but even understanding the semantic meaning of the image does not require strong segmentation. A weaker version of segmentation has been introduced in content-based retrieval. In weak segmentation the result is a homogeneous region by some criterion, but not necessarily covering the complete object silhouette. It results in a blobby description of objects rather than a precise segmentation. Salient features of the weak segments might capture the essential information of the object in a nutshell. The extreme form of the weak segmentation is the selection of the perceptually most salient points as the ultimately efficient data reduction in the representation of an object quite likely drawing human focus-of-attention.

Whenever the image itself permits an obvious interpretation, the ideal content-based system should employ that information. A strong semantic interpretation occurs when a sign can be positively identified in the image. This is rarely the case due to the large variety of signs. As data sets grow big and the processing power matches that growth, the opportunity arises to learn rather than to know the signs. One type of approach is appearance based modeling, learning from examples. That works but only if the recording conditions are highly standardized, like frontal well illuminated faces only. A better approach is one-class classifiers, from examples carefully describing the limits of a class of objects [Tieu, Tax]. An interesting technique to bridge the gap between textual and pictorial descriptions is to exploit information is called latent semantic indexing [Sclaroff]. The search is for hidden correlates of features and captions. In a broad class of images and the enormity of the task to define a reliable detection algorithm for each of them.

Similarity is an interpretation of the image based on the difference between two elements or groups of elements. Any information the user can provide in the search process should be employed to provide the rich context required in establishing the meaning of a picture. The interaction should form an integral component in any image retrieval system, rather than a last resort when the automatic methods fail. Already at the start, interaction can play an important role. Most of current systems perform query space initialization irrespective of whether a target search, a category search, or an associative search is requested. But the fact of the matter is that the set of appropriate features and the similarity function depend on the user goal. Asking the user for the required invariance, yields a solution for a specific form of target search. For category search and associative search the user-driven initialization of query space is still an open issue. We make a pledge for human-based similarity rather than general similarity.

User interaction in image retrieval has, however, some different characteristics from text retrieval. There is no sensory gap and the semantic gap from keywords to full text in text retrieval is of a different nature. No translation is needed from keywords to pictorial elements. We identify six query classes: exact and approximate and ranging from spatial content, image, image groups, and all combinations thereof. When accessing spatial content, some form of (weak) segmentation is required. A balance has to be found between flexibility on the user side and scalability on the system side. Query by image example has been researched most thoroughly, but a single image is only suited when another image of the same object(s) is the aim of the search. In other cases there is simply not sufficient context. Queries based on groups as well as techniques for prior identification of groups in data sets are still promising lines of research. Such group-based approaches have the potential to partially bridge the semantic gap while leaving room for efficient solutions.

A critical point in the advancement of content-based retrieval is the sensory and semantic gaps. The size of the sensory gap between the image data and the computer processed description - for a discussion see above - is enormous but in recent years some of its structure has become better known and even some partial fillings have been achieved. The size of the semantic gap - the distance between the image and the immediate understanding of a user not available to a machine - for a discussion see above - is formidable. The scientific progress is interesting but negligible to the size of the gap. Use of content-based retrieval for semantic browsing in general domains will not be within the grasp of the general public as they are accustomed to rely on the immediate semantic imprint the moment they see an image, and they expect a computer to do the same. Specific situations may work. The aim of content-based retrieval systems must be to provide maximum support in bridging the semantic gap between the simplicity of available visual features and the richness of the user semantics. The best way to resolve the semantic gap comes from sources outside the image by integrating other sources of information about the image in the query. Information about an image can come from a number of different sources: the image content, labels attached to the image, images embedded in a text, and so on. We still have very primitive ways of integrating this information in order to optimize access to images. Among these, the integration of natural language processing and computer vision should come first.

Nybble-engineering refers to a method that starts at the programming level of real-time tools. Here it is primarily their replicating components that represent a technical artistic challenge. The AV products generated from these, so-called nybble-engine-movies (NEMs), are structurally interactive real-time network films. Nybble-engine thus stands for both the entire program framework and for the action-bots in their function as non-linear emergences. In addition, a freeze frame is extracted on the basis of nybble engineering and sintered via a '3D printer': the data-objectile. This object represents a functional equivalent that is not analogously conceptualized, but rather in terms of problem orientation. The nybble-engine-project is generally presented in a pop-coded form as a live modular lecture.

The Tetrad is a conventional idea in the natural sciences. It must be imagined processually as a coiled rotation. For this reason, the tetrad is highly suitable for spatializing the theories intended to 'depict' a self-organizing system. As soon as a system begins to modify that from which it has emerged, as soon as it organizes itself, the unambiguous cause-effect relationship becomes a confusing cycle, in which the beginning is not only the beginning, but also a derived, subsequent entity, beginning and result, multiply allocated, without orientation, coiled, twisted, altogether complex. The situation represented here also describes the framework conditions for dealing with non-linear software tools, as in the case of the nybble-engine-project.

nybble engineering

An aesthetic message is usually the deconstruction of a conventionalized text form or a media text. It is recoded by destroying semantic portions in order to increase the aesthetic information. Nybble engineering applies this method at a basal level. This is less a matter of the interrelationship between semantic and aesthetic information, than the interrelation between uninterpreted and interpreted programs.

To better understand this, let us look at an example from biology. The notion of a living cell as an automaton, a machine, has come to be taken entirely for granted in scientific thinking today. So-called replicases, for example, corresponding to a duplication apparatus, use programs in two ways. In terms of molecular genetics, a distinction is made between transcription and translation. In one case it is a command resulting in a sequence of activities: the program manipulates the machine. In the other case, as an object it is passively subjected to the routines: the program is manipulated by the machine. Thus when the program is uninterpreted, it is merely raw material for the duplication. When it is interpreted, on the other hand, the program controls this duplication. On the one hand, the behavior of any machine can be formally described with a list of transition rules, and on the other hand, any list of this kind can also be understood as a potential machine.

First of all, nybble engineering needs existing material, to which the engineering can be applied. In the case at hand, this is real-time tools, such as so-called first-person shooter games, which represent a technical, artistic challenge with both their 3D surfaces and their replicating program components.

Structurally, nybble engine movies (NEMs) are real-time network-movies, AV-products resulting from nybble engineering. NEMs are based on certain program configurations that process, when they are executed, between interpreted and non-interpreted components, and are calculated on the respective human/machine interface (PC) in real time. The NEM is an interactive program, the course of which can be influenced by players and bots equally; it can be used as a multiplayer environment. The nybble engine thus stands for both the entire program framework and the action bots in their function as replicating program elements. A NEM show a demo run through non-linear program architecture, a kind of digital road movie.

Our understanding of the term 'machine' is oriented to its current state of differentiation and must always be considered in conjunction with the term 'coupling'. Norbert Wiener introduced this term with his ground-breaking work on cybernetics, published in 1948, and it is a formalism that makes it possible to link previously incompatible orders of the micro and macro level. Since then, the term 'coupling' has grown tremendously in complexity and differentiation with Second Order Cybernetics, General Systems Theory and Radical Constructivism.

When the tetrad is considered within the given framework of structural coupling, it is logical that the various generative factors, which are based not only on biogenic but also on technical programs, have an impact in socio-technical ensembles, starting at the program level, for instance action bots, game engines, authorship of the machine, continuing at the level of specific social contexts, as in the case of the 'modular lecture', which is executed in the interrelations of different media levels.

data objectile/Problem Orientation Instead of Analogy

The data objectile is determined by complex factors. Its physical presence not only stands for 'stable substance', but also for an unstable set of interactions, which are processed under the aspect of a multi-level conception (Varela 1990; Dupuy/Varela 1991) through self-descriptions, self-observations, and self-amplifications. And as with every articulation of art, it is a viewer arrangement, in which, in this case, humans interact with humans, machines with machines and humans with machines. At a certain degree of abstraction, it may be also said that trivial and non-trivial machines interact with one another. Generalizations, however, often tend to deteriorate into banality. In order to avoid this, the selected method must not be oriented at the most general level to characteristics or to differences in genre or to similarities, but rather to problems.

First of all, it should be stressed that the data objectile results from the data generated through nybble engineering. To be more specific: it is formed through a data extract and sintered via a 3D printer (SLS method, Selective Laser Sintering). It is a 3-dimensional 'freeze frame' of a real-time environment. The data objectile consequently has no reference of any kind to an object in real life. As the reference to problem orientation already indicates, the data objectile is not developed by way of analogy. Instead it is based on considerations that, contrary to classical subject-object epistemology, seek to negotiate a far more complex understanding of the object. This opens up the option of a functional equivalent that is generated in the reference to software objects and their implications in a dynamic, complex context. This, however, requires the terms 'specification', 'generalization' and 'respecification'.

Like every social context, art also evolves conventionalized expectations, orders and concomitant norms. With a problem orientation, we can specify the norm more clearly by specifying the reference problem as a risk of generalization that is immanent to meaning. Shifting the norm concept to the concept of generalization puts the data objectile into a more complex understanding of the object. This is already achieved through its status of being an articulation in a communication context, where expectations always make up an important factor. These are expectations, in other words, that are guided by norms and are, to a certain extent, dependent on a factual event; thus they are 'generalized expectations', which leave the content of what exactly is expected more or less undefined.

Although generalization stands for indeterminacy, it does not stand for the unknown, because what defines its functionality, is that it requires respecification in order to reach reference points. In this way, it is possible to evoke an interest in intensification, which addresses cognitive expectations and thus makes generalization a precondition for learning. Of course, it is also true in this case that one must know something in order to acquire knowledge. For it is only in an open combination of knowledge contents that remain constant and the willingness to modify and alter knowledge that generalized cognitive expectations are treated as knowledge.

With the definition of understanding art as an interdisciplinary coalition under specific system-immanent logics, the data objectile is targeted to the dimension of cognitive expectation in conjunction with specification, generalization and respecification. And as it is in science, the proportion of indeterminacy is made productive through the requirements of theory and method and thus referred to structures, which only apply to a specially differentiated functional system in society: in this case, the art system.

In the selected method, relating problems to one another is not an end in itself, but rather serves as primary orientation in seeking possibilities, or more precisely, seeking functional equivalents. This approach makes sense, when we define the historical art context as a relation of system problems or as a way of relating them to one another. The so-called anomy of (modern) art is only one striking example of this. In fact, however, every change of paradigms only represents the establishment of another ('new') function installation in the art system. In terms of system theory, this is part of the differentiation of function installations. In art discourse, the 'new paradigm' stands for a problem construction, to which further constructions - in other words, functional equivalents - are conjoined. Orientation to function tends with appropriate complexity to higher problem specification. This applies not only to the art system as whole, with all its subsystems, which are structured through orientation to function, but also at the level of art praxis, which articulates/topicalizes/reflects/problematizes these system/environment differences.

The employed method represents a special horizon of the world of life for specific intentions, which places that which happens whenever information is processed - specifically the probing of differences - under certain conditions, thus putting them into a certain form. From this perspective, the data objectile is a functional equivalent. Its functional equivalent relationship, however, does not only refer to the source text (the nybble engineering data extract) that gives it its physical form. Rather, the data objectile circulates in a functionally equivalent relationship to a cluster of art discourse types and mutual system/environment relations, which have become differentiated in the course of the historical art context in the art system. This representation is based on a premise of the General System Theory, namely that every social contact is to be understood as a system. What is essential to the functional method is its orientation to the problem of complexity and not at all to the problem of maintaining what is already there. This is simultaneously its claim to the preservation of complexity. Since the functional method is self-referentially founded in its basic complexity preservation, it identifies its semantic forms as problems of reference, which only follow their own system logic. This means that they do not mirror constructions of other realities, but rather seek to understand an ordered social reality as a form of order in relation to this social reality.

The data objectile does not rely on a principle of l'art pour l'art. Essentially it is socio-technical ensembles of humans and machines that form current realities. Biogenic programs, which virtually regulate the operation of agency in the constitutional context of person, role and expectation, conjoin with programs of a technical nature. Survival means preserving complexity in a cybernetic media world.

The methodological framework of orientation for the nybble engine project is a radical/re-constructivist meta-art.
'Meta-art' (cf. Art & Language 1972) is an evolutionary achievement of the art system, and its emergence is inseparably linked with aesthetic Modernism. In its final evolutionary stage, meta-art is a reflection-theoretical endeavor as a form for observations, in order to observe oneself as a second-order observer. This kind of art marks the end of the aesthetic era of the self-description of the art system.

In order for the communication of the form to succeed, however, the socially constituted expectations of art must be taken into consideration in terms of the aspect of viability (following Ernst von Glasersfeld). This means that a reflection-theoretical art that claims to preserve complexity must still articulate itself in forms that are still identifiable as art. In this way, meta-art processes along the system/environment difference, at the very edge, so to speak, within which the intensification of art's ability to dissolve and recombine can still reproduce art. It is not the aim of meta-art to explore existence in any form for the purpose of attaining insights with artistic means and methods, but rather to discursivize questions that relate to the how and the viability of art in its respectively current social context.

Of course, not every developmental step proves to be viable. "A sociology that regards modern society as a functionally differentiated social system, does not maintain that all functions succeed equally well when functionally differentiated. It has doubts with regards to religion, and with regard to art it can raise the question, whether differentiation is beneficial to this functional area..." (Luhmann 1984).

The current approaches of 'meta-art' only have little in common now with those of the sixties (cf. Art & Language 1972). The forms of current meta-art are now completely integrated in the post-autonomous art that has been developing for about twenty years. Just as the orientation to function serves both self-simplification and evolving complexity, the autonomy of art can be both weakened and strengthened through heteronomy. From the perspective of differentiation theory, the issue of autonomy and heteronomy is never an either/or question. Rather, both of these terms are conceived as a relationship of mutual escalation, whereby differentiated systems combine an increase of autonomy with a greater amount of dependency on a plurality of other systems.

The tendency of art in the last twenty years sufficiently affirms that pure, non-purposeful art no longer offers a motivational basis for aesthetic agency. An idea of this kind is no longer tenable in the plurality of an (ultra-) modern society. In contemporary art, therefore, it is much more a matter of initiating communicative fields of production via articulations. Under the 'frame phenomenon' art, these fields create various horizons of meaning for as many participants as possible. From this perspective, it is understandable that the nybble-engine-projects integrates elements borrowed from the dramaturgy of marketing and accordingly has three pillars: 1. the 'theme phenomenon', calling to mind 'narratives' from science, for instance; 2. the 'frame phenomenon', which is clearly given here through the art reference; and 3. the 'ensemble phenomenon', which provides for the participants to retain ensemble status beyond mere reception.

More information can be found on http://www.climax.at/

The work polar could be described as an interactive multimedia installation, as it was also described for the entry in the Prix Ars Electronica competition, but its complexity in a way defies this classification.

We envisioned the 7mx7mx4m totally connected and tactile space as a complex tactile matrix interface, that enables the visitor to experience the flow of data in the global and local networks in a completely immersive, yet cognitive way. The work was inspired by the notion of the cognitive ocean as described in Stanislaw Lem's and Andrey Tarkovsky's solaris. The initial conceptual equation was:

OCEAN = MATRIX.

The main outline of the work was based on the creation of two software and hardware engines, the so-called polar engine (with adjacent pols, polar dictionary and knowledge base) and the change engine (with adjacent trace route visualiser, which consisted of two separate displays). The first was envisioned as the input-analysis and construction zone, the second as the output-synthesis-experience zone. These zones were defined both in conceptual-software and hardware senses and also in the environmental sense. We have defined a zone in which the biological and physical, was directly interacting with the abstract-immaterial.

One main question was posed in this process:

How do we construct a cognitive and tactile experience of the seamless and near-abstract matrix with the analysis/construction/transformation of it included in the process?

Basically, we wanted to create an interface between the human body and senses and the matrix, which would by the sheer presence let alone the activities of this humans already transform also the structure of the matrix that is being observed/experienced and of the physical space that is being inhabited during this process.

the space

The polar space is a 7mx7mx4m white three dimensional physical space, which includes four projection surfaces, seven speakers, a 3D high resolution tracking system, a micro-organism growth module, a wave patterning water vibration module, two so called field displays-touch input modules and seven 'pol' zones. The hardware configuration of the space is clearly decodable. The whole construction was modular and includes also the ceiling, where a complex system of 32 lights controlled by the main engine of the project also played an active role in the tactile and sensorial experience. One of the main experiential goals of the space design was to minimize the presence of clearly discernible technological elements within it on one hand, and on the other to approach it from the pragmatic viewpoint of an all encompassing 'human-machine' interface.

the software systems

As it was previously described, the software systems were divided into two main zones. The first zone served as the input and cognitive zone, the second served as the output, sensorial and tactile zone. To connect the two a third routing module was created, named solaris. Its purpose was to route and connect all the systems in the polar network. To do so, a protocol to connect different software, hardware and OS systems was created. The protocol is one of the most invisible but also most important parts of the software development for this work, since it connected systems as diverse as the Canon Inc. polar modified Advanced Intelligent Information Retrieval System (AIIRS) and the Twosuns GMBH's Cartasia and Enclued software workbench and 3D tracker.

While the protocol was the language of polar the polar engine, polar dictionary and knowledge base were the heart of the content systems and the connecting point of the work with the information matrix. The knowledge base was built by the authors as the main initial content part of the work. The base consisted of at least 20 text files of 100KB for each of the seven categories for each language (the whole work is bilingual-English and Japanese) defined by the authors as the main content categories of the polar dictionary.

The seven categories were:

• crystal-crystallization
• diagram
• stealth-stealthy
• machine-machinic
• wave-waveform
• symmetry-symmetriad
• spectral-specter-spectrum

From this categories, the authors defined ten keywords for each of them, this 70 keywords in English and 70 keywords in Japanese were the starting point of the database search options for the users when they came to them during the exploration of the ocean space. The polar dictionary was enlarged during each session run, through the search of the matrix and the distillation of keywords from retrieved texts and documents.

The polar engine consists of three modules:

1. Keyword Distiller module
2. Knowledge Base Generator module
3. Category Filter module.

These modules select the useful information from the retrieved texts, summarize it, extract keywords and categorize it. The system used for this operation was developed from Canon Inc. proprietary software developed by the Media Technology Laboratory by the authors of polar. The multi-agent model called 'artificial staff' suited for the architecture of an information retrieval system was used. 'artificial staff' agents are autonomous, observe the users patterns and other agents and exchange the acquired information under a loose protocol. The flow-type information retrieved form the matrix is thus processed and categorized. The polar dictionary grows in an autonomous way, determined only by the initial users choice pattern and the result from the information retrieval and analysis from the texts based in the matrix. During this processes an immense amount of numerical data is created. This data was analyzed and created the main content transfer between the different modules in the polar network. This data was the blood flow of the system and was routed through solaris to the tactile zone and systems. They're the physical and tactile interaction with the visitors was of paramount importance.

The hearts of the tactile zone were the change engine and the trace route, data flow and 3D space visualisers. The change engine consisted of the Enclued and Cartasia software modules, which served as real time dynamic multi-media process generators and trackers. This was the output generator, which generated tactile and sensorial (light, space, sound) changes in the ocean. Along with them, a pair of displays showed real time data flow within the system and the matrix, combined with a 3D display of the space, so the displacement of data in the matrix triggered by movement in the space was visible. The second display was a two-zone trace routing display system, showing real-time trace routing data in Japan on the top and the world on the bottom part. The software for the data flow display was based on a latex wave pattern model and was written using OPEN GL and a proprietary library. This tactile zone was enhanced using other, analogue systems of interaction, that did not include software, but are described in the hardware systems part and experience flow part of this document.

the hardware systems

The hardware systems consisted of a combination of computers connected in a LAN, a MIDI connection for the light systems, two mobile wireless computing units (pol), display systems and analogue loop systems. The systems were connected through a protocol and also by analogue lines. They included also a micro camera filming the process of growth of the human cell, which was triggered by the temperature changes connected with the usage of the space and a system of an analogue input of sound and wave pattern data into two projectors. The third analogue system used was the wave patterning water vibration module, which consisted of a special fitted pool of water, connected to two subwoofer bass speakers, that changed wave patterns in it. For the connection between the LAN digital systems and some of the output analogue systems sound was used as the main trigger.

The four projectors used in the system showed:

• data flow and 3D real time space situation
• trace routing
• analogue output and changes of a test pattern video system triggered by sound inputs
• magnified picture of a human ear cell in growth in the space during the two weeks of operations

The other hardware elements of the system were the two field displays, which were the main user interfaces with the polar dictionary and the 'pol' units, which are two mobile wireless sensor packages, which collected and relayed temperature, acceleration, sound and visual data and generated the first set of numerical data, collected in space and by the user, to trigger power up of the field display and the initial generation of the keyword list on each of the visitors monitors. The digital hardware and computers were connected in a wireless and wired LAN, MIDI lines and analogue cables. During the exhibition, a web camera was placed in the system, that enabled users on the web to observe the situation in the space and the visitors exploration and behavior.

The 'pol' was the main interface of the user during the initial exploration of the ocean space. It included a sensor package measuring outside and inside temperature, a 3D-axis accelerometer, a still camera and a sound recording system. With the 'pol', the user explored the space. It was based on a SONY VAIO PCG-C1VN Transmeta Crusoe based computer, a wireless LAN card and the proprietary sensor package. The 'pol' was envisioned as the first encounter of the users with polar and a tool, which will generate a code sequence for the start of the change processes in space through physical exploration of this same space. When the 'pol' was placed in the right position in the space on one of the seven 'pol' zones, the space changing process started.

The 3D tracking of the visitors and the 'pol's' in space was done through a motion tracking system, consisting of six computers and twelve IR cameras, tracking four IR sources. The system has a very high resolution and was developed by Twosuns GMBH for polar. This tracking system constantly relayed positioning data back to the main solaris router and the change engine. This data was included in the main routing and accordingly, the changes in space were generated.

experience flow

With interactive works it is of course very difficult to describe the experience flow of them, especially in ones, that like polar are constantly changing, but we will give it a try. We will do this in two parts, one describing the visitor's point of view and one describing the data package point of view.

Visitor As the pair of visitors come in front of the space, they are explained the main rules of the interaction within the ocean space. The functioning of the 'pol' and the users interface of the field display are also explained to them by the hosts. After that, they wear a special jacket, which includes the visitor's infrared ID badge. Then they are given each one of the 'pol's' and they enter the space. In the space, as it is also clearly audible in the video documentation, there is a constant low hertz hum. This hum changes immediately as the visitors enter it. The system senses the 'pol' and the visitor and starts reacting. The light patterns start to change very slowly, the sound also and the temperature in space too. These processes of space-time change are based on a random generation of data, combining random generators with the real time data flow from the 3D tracking system and the 'pol' data output.

The visitor has two to three minutes for this initial exploration, then a specific sound is heard, that marks the point where a special place for the 'pol' must be found. The 'pol' must then be placed into one of the seven 'pol' zones. When the right zone is chosen, the space reacts with a very bright and quick flash and a totally different high-pitched sound. The 'pol' is in the right position, and the random keyword choosing sequence is generated from 'pol' gathered data, the input field displays are then turned on. The visitor much chooses the language of interaction and then the chosen keywords are displayed. The solaris sends a message to all modules that the change processes have started. When one of the visitors chooses a keyword and places it in the system, the solaris sends the keyword to the polar engine. Exactly what happens with this keyword will be explained in the data package point of view that follows.

For the visitor, the time lag between the involvement of the keyword in the matrix and the first results is the time in which he can and has to explore the ocean space. This exploration changes the position of sound fields in the 3D configuration of space and the initial numerical results from the 'pol' data and now also from the matrix search processes change different sound pattern levels. Because of this interrelation, not two experiences in the space are alike. The visitor can observe the wave patterns changing as a result of his movement and movement of data in the water module, he can observe the cell organism live and projected, the temperature in the cell module also fluctuates according to data generated through movement and through the matrix searching. Of course, only a visitor on the first and last day could see the difference in the actual growth of the cell...

The data flow and 3D tracking combined wave display make the user aware that the space she/he inhabits is also a data space, since data flow patterns interact with the patterns of his position and the position of the 'pol' in the space. When the trace routing display starts (very late in the process, because of the large amount of data from the tracing that needs to be processed and parsed), usually the field display is ready for another keyword input. The list of keywords of course grew and a new keyword list is in the meantime generated. The visitors had a chance to involve two to four keywords in one session, depending on the keyword and the result base. This was because the time for the exploration was limited to 10 minutes for two visitors at a time. Of course, longer times can be set for the exploration and thus more changes in the space-time data-space could be observed.

After the clock runs out, the space automatically turns the lights on at maximum and the process of cooling down is started. This process enables the visitor to understand that the change process is over. Also on the field display the visitor is reminded that the process is over and that she/he must leave the space.

Data package The experience of the data flow in the processes in polar is as hard to describe as the experience of the visitor, since there are many flows of packages at once. But the basic pattern is as follows:

• 'Pol' acquires environmental data, acceleration, temperatures, visual data, sound
• When 'pol' is in the space, the 3D tracker starts to track the 'pol' and the visitor and relays the tracking data to solaris
• This tracking results in the light pattern and sound pattern changes in space. The space is full of light/sound positions that are constantly shifting according to the data input form the tracking system
• The system has a time limit and 'pol' stops collecting data after two to three minutes
• The tracking system waits until the 'pol' is put into the right position, randomly chosen from the seven possibilities according to the 'pol' data
• The whole package of the 'pol' data is sent to solaris
• solaris sends start commands to other modules
• The field displays start up and wait for the language input
• When the language is input, the polar engine chooses a list of seven keywords from the polar dictionary
• The keywords are displayed on the field displays
• The visitor chooses a keyword and when he is placed in the right position the keyword is sent to solaris. In all this time, solaris relays the positioning data and the data flow information to the data flow visualiser, which was changed from sound visualiser in the scheme.
• solaris sends the chosen keyword to the polar engine, which then sends the keyword to 15 different search engines for each language
• The results pages from the search are put into a temporary data pool, these results are then analyzed, compared to the knowledge base and the new possible keywords are distilled. The results are also analyzed for the URL, and trace routing. Each URL of a positive result is trace routed and this data is then sent to the trace route visualiser.
• All the numerical data generated in these processes is sent to the change engine through the protocol and the space-time-sound situation in the space is constantly shifting.
• After the keyword is added to the polar dictionary, the solaris sends a message to the field display that the time is right for another search
• The process could continue indefinitely, but a timer in the solaris can be set and is in an exhibition situation to a limit, where the processes of one session end.
• The systems and the modules power down and wait for the new session.
• The transformation is completed.

Conclusion

As it was said in the beginning, we posed ourselves one basic question:

How do we construct a cognitive and tactile experience of the seamless and near-abstract matrix with the analysis/construction/transformation of it included in the process?

With this system, we are trying to answer to this question. Also, as said in the beginning, the ocean from solaris was the inspiration for it, with its complexity and cognitive elementarily. We wanted to create such and experience of an ocean similar to that in solaris, but a different one in the sense that it combines data and topology of the matrix with a real 3D space-time in an all encompassing tactile experience. It tries to materialize the immaterial. For us, one of the main goals was that each interaction of humans-visitors with this data and physical space, would result in it being different than before, transformed in size and shape.

Sounds within the data space shift, sound and light trigger positions too. New are generated in the process, and the dictionary, based on the seven basic categories set up by the authors too. Everything is shifting and changing, some things are growing (cell), some physically changing (water).

polar has a life of its own.

More information can be found on http://www.canon.co.jp/cast/artlab/artlab10

Recognition is definitely a part of visual intelligence. It relates an emotion, experience or visual input to an earlier event. Image recognition is based on having seen something before. Our brain is effective in this. Human beings are able to understand and work with concepts. Computers are far less intelligent.

To a certain extent concepts can be expressed in language. Language enables the description of concepts; we are able to explain what we see. Although it is a tedious task, someone escorting a blind person can describe the things he sees to him. Even harder it is for the blind person to imagine what has been described. The mapping from image space to concepts is not one-to-one, after all: "A picture may be worth a thousand words".

In contemporary multimedia applications, an image is described by features, like the color or shape of an object. This is a many to one mapping and as a consequence, there is no one-to-one reverse mapping. A car can be red. But there are a lot of other objects with a red color.

As we can learn from the human seeing-eye dog, one way of doing is to describe the content in language, as done in the case of keywords. But then a more intelligent way of looking for similarities is required. 'Visual information retrieval' (VIR) systems process visual content in a way human beings do. 'Content based image retrieval' (CBIR) is based on the fact that images can be retrieved because of their similarity to other images.

One can distinguish three core components of these systems:

• Content extraction
  Describe the content in such a way that it can be processed. 'Feature extraction' is one way of doing this. An image or video is described according to several features like color or texture.
• Similarity
  Once the content has been described, the system has to define how similar this content is to the content of other images. As a result, one has to define metrics approximating to what extent the different images, represented by their features are similar.
• Interfaces
  For the user to communicate with the system, interfaces should be able to display and compose visual information. As the content of images is subjective, an intelligent system should be able to manage this subjectivity.

As we are living in the age of information technology, with the amount of content growing exponentially, the need to handle and archive this information grows as fast. We are overwhelmed with information:

http://www.sims.berkeley.edu/how-much-info

and the ability to retrieve and archive this information becomes proportionally smaller to the growing amount of data. In the domain of 'visual information systems', information can be processed for several purposes:

• Compression
  MPEG 4, 7, 21 add standards for describing content in multimedia databases, besides mere compression.
• Retrieval
  To browse, query and download information from the web or other information databases.
• Visualization
  In the field of fashion and design for instance, fashion depends on personal appreciation. Applications should give way to examine clothing as it appears ('intelligent clothing') and search for products by visual means.
• Security and authentication
  Document ownership, access and facilitation. Authorizing or blocking content (pornography). Filtering.
• Quality control
  Visual inspection of products like textiles.
• Delivery on demand
  Personal Content as in television on demand. Extracting personal content from a larger and more general content quantity. Indexing.
• Manipulation
  Content being processed in a way that new content is created. Examples are art, music and video. Tasks: archiving, billing, accounting, virtual shops, etc.

Michael:
Perhaps the first thing to say is that there is nothing about the concept of archive that insists that there is public access, despite the fact that as a location it is the place where public records and historical documents are kept. The archive, like a court of law, does its job independently of intimate public participation. Secondly, in spite of its image as an unworldly apolitical bastion of objective history, the archive (in the modern sense of the word) stems from precisely the time when the cult of fact was superseding other ways of knowing the truth. Regardless of its content, the modern archive (that is, post seventeenth century and pre-Postmodern) may contain evidence of the subjective and the historical, but it is at pains to eradicate the contamination of subjectivity and history from its methods. Such contradictions insist, at the start, that the archive, for all its apparent timeless stability, is a site of complex contradiction and a site of political activity. Restricting public access to those who subscribed to empiricism and what has been called 'boundary maintenance' ensured that these contradictions were stabilized.

The premise of this conference (as I understand it) is that wider public access, facilitated in part by certain technologies and in part by social revisionism, has placed the archive (as a concept) under strain in such a way that its disciplining function is either exposed or eliminated. Certainty, whose supremacy is evaporating in history as much as it is in cosmology or television, needs a worthy usurper that will reinforce the values that underlie the cult of fact. Into this moment of instability a number of artists and scientists have seen new opportunities to reclaim the real by building alternatives. It remains to be seen how long the space is available for us to act and what scope there is for real intervention. Nonetheless in the secessionist spirit I ask the question intended to foreground the issues of subjectivity and history: Can you have a beautiful archive?

Anne:
Do you mean storage wise or in terms of retrieval? It is an interesting thought in the digital age - it somehow seems to refer to the physical presence of let's say a library - objects and the atmosphere attached to them, multiple sensory input like the smell and feel one can experience in 'special' collections or personal archives. Software code and d-base ontologies sometimes possess this beauty and 'personal signature' as well. It also refers to the importance of graphical user interface design, which is an important field artists have been working in. In my talks to Ben Schouten he also refers to a certain aspect of beauty in visualization and design technology.

Michael:
I mean in the sense that the post-historical archive lacks the nostalgic sentimentality of the socially authenticated document and replaces it with evidence of the movement of history back and forth.

Anne:
Do you refer to history in general or to the history of objects, traces of usage, social or cultural history?

Michael:
I mean the use of history as a means of explaining the present, which for many people presents insurmountable contradictions. The authentication of data and the cult of fact is a relatively modern invention. We should not lose sight of the prior models of truth that the challenge of post-history recuperates.

Anne:
Yes, here we go! Although I always have my doubts about the term 'truth' as such.

Michael:
I mean truth in the sense of the competition between realities in which a reality informed by reason is claimed as the only admissible reality. We should not forget the popular resistance to reason in the past, or overlook it in the present. The crisis in the new archive in which public (and artistic) access and participation is irresistible is that the contrast between the decaying system (of civilizations) and the complete and generative archive is exposed.

Anne:
Yes, this also hooks up with the autopoietic system. After I studied the work/papers/writings of Thecla Schiphorst and Margarete Jahrmann in more detail I think it is interesting to keep in mind - while investigating the other works - for discussion and conversation/interview purposes:

• Thecla Schiphorst refers to Varela. In her work this may be interpreted as 'open' systems emerging/growing or organizing themselves through user interaction and social interaction.
• Margarete Jahrmann refers to Luhman's social system theory (Very briefly: he takes the communication domain as the term of condition for the autopoietic system derived from Maturana/Varela. From this theory one could conclude that the operational characteristics of his system theory are independent of its specific users/participants, since Luhman filtered out humans from his model.)

I would like to know how this relates to the self learning systems used by Ben Schouten and by Arnold Smeulders (refering to M. Minsky's society of minds). Distributed knowledge and distributed consciousness relate to this as well.

Michael:
I think that the introduction of debates about consciousness is essential. The electronic archive now reinstates these debates as it presents a challenge to the hierarchy of evidence, i.e. bookish literacy and mathematical data, which insist on certain interpretations versus the image, which thrives on an indifference to reasonable truth. In her brilliant book on visual analogy Barbara Maria Stafford points out that "Through sheer skill, craft, or technical trickery, poet and painter were able to imaginatively simulate mythical, historical, or religious scenes they had never in fact beheld. Conjuring and artistry were thus one."

This moral promiscuity invests the image with an extraordinary power in networked communications since its meanings are relatively non-hierarchical and immediate. Moreover the completeness of the image in contrast to the historical moment's decay raises new issues about the status of the image as archaeological and archival evidence.

Anne:
Yes, the image-less art (the art without fixed or attached image). This makes sense of constructions of images based on data retrieval and interpretation, where an image or a series of images is continuously being constructed.

I am fascinated by concepts of generating archives, not reflecting existing files, images or data to be preserved but to look upon archives as living organisms. This not only reflects other visions on archival concepts but also includes pointers to our contemporary life and approaches of art based on parameters, or 'intelligent' software code.

Michael:
As a complete catalogue of an incomplete past the archive becomes an archaeological trace of a fractured unconscious which finds expression in technologies that disavow the immaterial as they open access routes to it. In this way the archive can become a site of therapeutic retrieval regardless of its evidential base. In this case how (and with what technology) it is catalogued is as significant as what is catalogued.

I think that what we have to be aware of is the powerful inertia of the archive as an instrument and if we want to make interventions we have to be vigilant. For example we should be constantly asking to what extent our interventions in what we may begin to understand as the post-historical, post mechanical, archive merely recycle the aesthetics of High Romanticism and the mythologies of the cult of fact? I suppose to answer my initial question, a beautiful archive is now possible for the first time but if we achieved it then we should be suspicious.

Anne:
This reminds me of something I ran into at your webpages http://www.stem-arts.com/postdigital/ page1.htm

I found a picture with the following text: 'the imagination is prompted by human desire to modify the world through technology which in turn prompts desire'

I guess we're much further now. We are getting techno-morphed, in A.I. and robotics theory there doesn't really exist a single 'self' anymore. Most of the developments of the past few decades are heading in the direction of distributed knowledge. A while ago V2_ published a book called Technomorphica, which deals with technomorphization, the reorganization of the organic based on the intelligent machine model. It poses the question if evolution has entered a technological-scientific phase where humans no longer develop themselves in natural processes, but where the human body adapts itself to the parameters of this technological era. The same subject is reflected in the work of the artists and scientists involved, striving toward technological perfection. Artificial intelligence as referred to by M. Minsky in Machinery of Consciousness for several artists is a reason to come up with other scenarios to mis-use or abuse technology, to establish other cultural experiences or raise awareness among a broader (networked) audience and this is often done by means of connection machines.

Michael:
However, I would say that in your last comments it is precisely where I think we need to re-think terms such as mis-use and abuse since they suggest a legitimate use for technology that is in some way inherent in the apparatus. Its a position I find difficult to sustain since, as I argued in Accidental Machines, much technology - especially computers - was not 'invented' with an end use in mind. Rather, a particular meaning became validated by a forceful colonization by the very community that had the most to lose from alternative interpretations. What we now see with artists, hackers and gamers is the trace of those other meanings, which are simultaneously exploited and denigrated by the 'legitimate' users. In the same way, if we regard the archive as one of the technologies of history we return to the opening statement and the archive's vampiric relation to history as concept feeding off its own species while at the same time claiming to occupy a territory regarded as aloof. Of course we are talking generically here and not about any particular archive, and that has to be stressed, because what is at stake is the meaning of the past as mutable negotiable consensus between equal partners, not an absolute imposed by a series of institutional gateways whose invisibility is an illusion in which we all conspire.

Note:

This conversation could have been carried on for a while. The above text is a reflection of some parts of our discussion. Publishing this text in the conference reader should not be interpreted as a total coverage of the theme. Nevertheless, since we are talking about archives it seemed relevant to open up our conversation archive created around the theme of this conference. We prefer to deepen some of the issues discussed during the conference Anarchives: connection-machines July 5th 2002 with the participants and the audience.

Media are increasingly becoming means to construct realities, rather than means to represent reality. Media do not depict the existing world, they replace it with a different reality, a 'media reality' or rather 'media realities'. How different media shape reality is a recurrent theme in V2_ programs.

During 2002 and up to and including DEAF03 V2_'s program focuses on the political, economical, social, historical, epistemological and software-based implications of techniques for data clustering and data combination. The program emphasizes the role of interactivity as a method to manipulate, transform and individually shape media realities. V2_ takes a fundamental interest in the present; a present which condenses into an archive in front of our very eyes.

In various contemporary views the archive has proved to be a strong metaphor. Our body has become a genetic archive, now that it has been digitally opened up in the Human Genome Project. Culture, as found in museums and other art institutes and in magazines and cultural supplements as well, has been described as an archive that new art both has to react against in order to be new and has to penetrate in order to be recognized as art. Our language is an archive of meanings that can be unlocked by philological methods. It teaches us who we are and where we come from. The unconscious is an archive of all the traumatic experiences that define our identity. History is a database from which facts can be arbitrarily retrieved, and now lacks a unifying story.

It could be argued that the latter post-modernist view on 'fragmentation' and 'deconstruction' of everything in the archive of history is related to the way in which data was stored and retrieved in the early days of the information age: via non-hierarchical, non-linear search engines. Digitally speaking, all data were equal then, whether they were text, image, sound, protocol, program code or whatever. Since the 1990s a new form of structuring digital archives has emerged. Now it is not just the individual data that are being stored in databases. The relationships and correlations between the various data are now also being stored, by using 'metadata'. Metadata (also known as 'tags') are data that describe and categorize other data. Metadata as means for ordering, hierarchizing, streamlining and evaluating have become increasingly important as social, political and economical instruments in what has been considered an informational sphere free of values for so long.

Information isn't power, but knowledge is. Knowledge is tagged, or intelligently grouped and combined, information. Knowledge is the result of the (private or public, controllable or associative, open or concealed) knowledge management of data and data clusters.

Archives no longer just hold our past for inspection by historians, tax collectors and other researchers. We are permanently living in archives: all the sites we visit on the Internet are being logged by our search engines. All our shopping is being registered by our supermarkets. Each time we perform an electronic act we add information to the running archive of our activities as both individuals and members of target groups. On the basis of such archives the policies for the future are being planned, from marketing strategies to decisions about where to build shops. Behind almost all activities in the hard, material world nowadays there is an immaterial archive, for instance the storage of data from video surveillance and other security equipment. We are living in the world's online archive, or, more to the point: we are living in the world-as-archive, as a constellation of databases.

Because they are continuously available and accessible, archives have become an essential factor in acting in the present. One could even say that archives have become crucial in how the present is created and reflected upon. Archives are becoming just as process-like in character as the present already is. The individual's experience of the present can be increasingly described as the moment when an 'unforeseen' link is forged between dated information clusters that reach him or her through the media. Why does an archive allow one connection to be established and not the other? Through which media and by what software can which connections be made within archives and between them? And which connections are excluded by those media and software? What role does the individual have to play in this?

Knowledge can be imposed upon the users of archives, or it can be developed by them by using strategic tools and agents. A growing number of artists, artists' groups and architects are developing (software based) systems in which data are organizing themselves into complex knowledge systems of which the users are but one of the organizing factors. Databases, software and archives are increasingly the objective of artistic interventions. How are the connections that archives do (or do not) make short-circuited and what is the nature of these short circuits? What is their artistic potential?

In the course of 2002 up to and including DEAF03 V2_ explores our media reality/realities in art projects, performances, lectures, workshops, Internet projects and cinematic screenings.

Architecture Fiction / Design Fiction

If design can be a way of creating material objects that help tell a story what kind of stories would it tell and in what style or genre? Might it be a kind of half-way between fact and fiction? Telling stories that appear real and legible, yet that are also speculating and extrapolating, or offering some sort of reflection on how things are, and how they might become something else? Design fiction as I am discussing it here is a conflation of design, science fact, and science fiction. It is a amalgamation of practices that together bends the expectations as to what each does on its own and ties them together into something new. It is a way of materializing ideas and speculations without the pragmatic curtailing that often happens when dead weights are fastened to the imagination. Design Fiction is a different genre of design. Not realism, but a genre that is forward looking, beyond incremental and makes an effort to explore new kinds of social interaction rituals. As much as science fact tells you what is and is not possible, design fiction understands constraints differently. Design fiction is about creative provocation, raising questions, innovation beyond the ‘up-and-to-the-right’ sort, and exploration. Design fiction works in the space between the arrogance of science fact, and the seriously playful imaginary of science fiction, making things that are both real and fake, but aware of the irony of the muddle – even claiming it as an advantage. It’s a design practice, first of all – because it makes no authority claims on the world, has no special stake in canonical truth; because it can work comfortably with the vernacular and pragmatic; because it has as part of its vocabulary the word ‘people’ (not ‘users’) and all that implies; because it can operate with wit and paradox and a critical stance. It assumes nothing about the future, except that there can be simultaneous futures, and multiple futures, and simultaneous-multiple futures – even an end to everything.

There’s a scene in the film Minority Report, which also happens to be a wonderful prototype of a ubiquitous computing future, in which Tom Cruise’s character Inspector John Anderton manipulates a database of sound and images that are from the near future. In this scene, which just about everyone in the world knows about, Cruise’s character makes orchestra conductor- like gestures, summoning and juxtaposing fuzzy snippets of what is almost about to happen. It’s all happening in a mad-dash effort to piece together a puzzle. The puzzle is, of course, unlocking the mystery of a murder we know will take place, unless the clues of its location and perpetrator are discovered. The example I bring up here is, of course, the gesture interface that Anderton uses to piece together the clue fragments for the future murder he is investigating. As a film element, it has a well-balanced mix of visual dynamics that will keep today’s science fiction film audience riveted, and legible interaction rituals that allow the audience to follow the gestures closely to develop an understanding of precisely what is going on – what is being manipulated and how bits of clue are juxtaposed and re-arranged as one might do so with a puzzle. Special attention is placed on the precision of the gestures that Anderton uses in order to manipulate the fragments of video and sound – zooming in on a bit of imagery with hand-over-hand gesture; deleting a few things by moving them with a forceful and dismissive sweep into this interface’s version of today’s user interface trash can. There’s more than the clue-construction device that Anderton uses – whatever its called. It would be a simple matter to show a few still images from this sequence as an index to the small bit of argument I’m presenting. But, it is precisely this longer bit of story that I want to highlight, and not just the instrumental technology. Not the story itself – the pre-murder. Rather, I want to highlight what the story does so as to fill out the meaning of the clue-construction device, to make it something legible despite its foreignness. It’s a device used to edit sound and images somehow extracted from the future. It’s as if the story is sharing with the audience, who may be reasonably wondering – how do you edit and manipulate fragments of sound and images from the future? How does police evidence gathering work in the year 2054, when evidence is things that have not yet happened – but will? Do they travel into the future through some device and collect things that they bring back? Do detectives still use little baggies and tweezers to collect scraps of bone fragment, sending them to clever forensic scientists back at the lab?

Science-fact and science-fiction are entangled in the Minority Report drama, which isn’t a bad thing. In fact, it should happen more. Science-fiction has way more imagination than science-fact and almost certainly circulates knowledge – wherever between fact and fiction that knowledge may live – and ideas more effectively than all the science journals and science journalism in the world. In the production of Minority Report, the idea for such a gestural interface came from somewhere and at least in part from the film’s technical consultant, John Underkoffler. Underkoffler was a member of the Tangible Media Group at M.I.T., and had participated along with a panel of luminaries in providing some speculations as to what the future of Minority Report might be experienced based on their insights and their extrapolations of the current trends in the technology world. What was needed were some futurist-style projections to help trace a vector from the speculations of the present to their materialization in the future of 2054, when the film takes place. From a project at the Tangible Media Group called ‘The Luminous Room’ were a number of ‘immersive’ computing concepts that were drawn from some of the principles of Ubicomp. The principles are related to the idea that computers might become more directly integrated into the architecture of the environments that people occupy. Rather than manipulating them with a keyboard and mouse, people might use gestures for direct input. Translating laboratory principles into a dramatic film allows for the lab ideas to circulate in a bold fashion, beyond what would be accepted in the typical, conservative research-academic-industrial context. There is a larger military-industrial-light-andmagic complex in effect here, which is precisely the larger, messy tangle through which fact and fiction become indistinguishable through a blend of science and entertainment. The action is a kind of science fact-fiction work that effectively tries out some ideas within a film’s narrative. It’s sort of like prototyping – sketching out possibilities by building things, wrapping them around a story and letting them play out as they might. More formally, this is what David A. Kirby calls the ‘diegetic prototype.’ [David A. Kirby, ‘Future is Now: Diegetic Prototypes and the Role of Popular Films in Generating Real-World Technological Development’ forthcoming in Social Studies of Science, a journal.] It’s a kind of technoscientific prototyping activity knotted to science fiction film production that emphasizes the circulation of knowledge and ideas. It is like a concept prototype, only with the added design fiction property of a story into which the prototype can play its part in a way different from a plain old demonstration. The prototype enlivens the narrative, moving the story forward while at the same time subtly working through the details of itself.

‘..scientists and engineers can also create realistic filmic images of ‘technological possibilities’ with the intention of reducing anxiety and stimulating desire in audiences to see potential technologies become realities. For scientists and engineers, the best way to jump start technical development is to produce a working prototype. Working prototypes, however, are time consuming, expensive and require initial funds. I argue in this essay that for technical advisors cinematic depictions of future technologies are actually ‘diegetic prototypes’ that demonstrate to large public audiences a technology’s need, benevolence, and viability. Diegetic prototypes have a major rhetorical advantage even over true prototypes: in the diegesis these technologies exist as ‘real’ objects that function properly and which people actually use.’ [Kirby]

The film becomes an opportunity to create a vision of the future but, perhaps more importantly, to share that vision to a large public audience. In specific cases, such as the evocative ‘gesture interface’ concepts Underkoffler introduced into the film’s story and its production design, ideas gather a kind of knowledge-mass. They become culturally legible and gain weight and currency. We ‘get’ the idea of using conductor-like gestures to interact with our information technology because it is given to us through the film, it’s pre-science, the discussions that evolve in media and with friends, the formation of companies to further develop the ideas, bolstered on the cultural literacy with touch and gesture interactions, and so on. To gain cultural legibility takes more than a scientist demonstrating an idea in a laboratory. What is needed is a broader, context — such as one that great storytellers and great filmmakers can put together into a popular film, with an engaging narrative and some cool gear. The follow-on to this science fiction film introduction of gesture interfaces to a large public audience are more gesture interfaces, each one staking out Minority Report as a point of conception, either explicitly or implicitly. It’s as if Minority Report serves as the conditions of possibility for more and further explorations of the possibility for gesture interaction — whether touchbased gestures, as in the Apple iPhone and other techniques, or free-space and tracking gesture interactions, like the Nintendo Wii, for example. This is not precisely the case: we are not interested in claims as to priority, ownership and who did what first. What is much more interesting is the brocade of activity that weaves in and through the fictional/factual special effects props of Minority Report.

The signs are everywhere. Ars Electronica's Infowar, ISEA's Revolution, Steirischer Herbst's Art and Global Media, Interpol's operation Cathedral (an international roundup of net pornographers), the launch of a 'satellite' over Japan by North Korea, 30,000 Iranian troops along the Afghan border, the Serbian siege of Kosovo (and the impending attack by US backed NATO forces), perverse media mergers, reactionary utopianisms, imploding economies, the return of distinctly undemocratic forms of repressive tolerance, the revealingly pathetic links between monogamy and monopoly, monotony and the fundamentalization of sexuality by the compulsive theologies of conspiracy - in short a world in crisis.

And if the phantasmatic information economy has provided a shield against the crumbling material world, one was reminded by George Stein at the Infowar symposium that 'information leads to dependency, dependency to vulnerability, vulnerability to defeat'. Not a rosy picture for the wired world. Yet while the promises of international information integration provide the fuel for strategic development, a shadow of fallibility emerges to rupture the systemic illusion that our social software will stabilize crisis. The mistake was innocently efficient and overwhelmingly problematic. Two digits ignored by programmers and hardware designers have posed more than a dilemma to a culture subservient to computers and their infallible memory for numbers. Rather than include 19 before the year in the 20th century, computational dates were indicated only by the last 2 digits. And now, 16 months before the millennial clock ticks to a new century, the 'time bomb' looms ahead as what Paul Virilio calls 'the integral accident'.

'Y2K,' said Alistair Cooke on a BBC report in June, 'probably the most ominous logo, the most threatening symbol to human life, since E=MC2.' This dazzlingly portentous comment was made with the calmest of demeanors, as if the destructive potential of the problem was as inevitable as it was surreally unmanageable. Cooke's report quoted heavily from US Senator Bennett's speeches to hearings on the Y2K issue that outline priorities and strategies for approaching the issue. Yet, doing much more than crisis management seems futile. Bennett writes that 'the world as a whole is almost doomed to have major problems because other countries are way behind us - however badly prepared we are - in their thinking and planning for Y2K'. The priorities are significant. Bennett outlines seven as essential: utilities, telecommunications, transport, financial systems, government services, business activities, litigation. Tests done with several power companies have not been favorable. As an experiment, two in Great Britain set their clocks ahead. At midnight on December 31st, 1999 both went off-line. Bennett indeed suggests that rather than attempt to fix the related problems in software and hardware, government agencies ought to prepare crisis management plans to stay in operation and to extend this readiness to the companies involved in essential services like the power grids, water systems, international air traffic controllers, etc.

Why 2K? Because, as Virilio has always reminded us, 'every technology brings a corresponding form of accident', because foreshortened assumptions about failure are blissfully ignored by cost-benefit analyses, because failure and progress (as this century has so dearly signified) go hand-in-hand as a measure of manageable catastrophe, because the adaptability of codes wrongly seemed a simple process of substitution.

So the crisis countdown continues. Jean Baudrillard characterizes it thus in a recent essay called The End of the Millennium or the Countdown: "For this century - which can do nothing more than count the seconds separating it from its end without being able, or really wanting, to measure up to that end - the digital clock on the Beaubourg Centre showing the countdown in millions of seconds is the perfect symbol. It illustrates the reversal of the whole of our modernity's relation to time. Time is no longer counted progressively, by addition, starting from an origin, but by subtraction, starting from the end. This is what happens with rocket launches or time bombs. And that end is no longer the symbolic endpoint of a history, but the mark of a zero sum, of a potential exhaustion. This is a perspective of entropy - by the exhausting of all possibilities - the perspective of counting down to infinity. We are no longer in the fatalistic, historical or providential vision, which was the vision of a world of progress and production. The final illusion of history, the final utopia of time no longer exists, since it is already registered there as something potentially accounted for, in digital time, just as mankind's finalities cease to exist at a point where they come to be registered in a genetic capital and solely in the biological perspective of the exploitation of the genome. When you count the seconds from the end, the fact is that everything is already at an end."

II.

Gödel was right to remind us that contradiction was an essential condition: no assumption without its opposite, no control without defeat, no seriousness without absurdity. So if fallibility characterises experience, then the whole absurd enterprise of militarised notions of interactivity (based on command and control strategies) is a myth hinged on the mystification of the misleading assumption that systems won't fail (or that their's will fail first!). Even the 'indestructible' and redundant Internet, impervious to nuclear attack, proves vulnerable not from external assault but from unintended error.

Mutations, accidents, blunders, oversights, omissions, faux pas - have, indeed, rotted many of the developments of the past generations. And if the jettisoning of infallibility can be usefully employed in creative ways, we might be able to rethink the algorithmic imperatives that envelop electronic media. Surrounded by pieties of all sorts - ontological fantasies, epistemological illusions, traumatized psychologies, anecdotal embodiments, over-dramatized technical reason - the field of electronic art has been cast as a sphere in which managed computational performance is sustained by extravagant allegories of exactitude, flawlessy debugged performance at the expense of the possibility of unpredictability. These mystifications compel an acknowledgement of imperfection, error, and, ultimately, failure.

This willing acceptance could easily be read as the admission of failure or even as a form of technophobia. Yet, as is clear, an assault on the 'triumphs' of reason, on the flaws in the system, can expose more than the imperfections of technology. They can extend the de facto normative, and blissfully functional ideologies of modernist technology into destabilized, ruptured, and absurd systems that are more laughable than logical, more reckless than rational, and perhaps more interesting than predictable.

© 1998 Timothy Druckrey / V2_

Globalisation as a general cultural and economic trend is a scam. There is no overall unification of the 'Global Village', and the 'New World Order' is a slogan used by those who would like to define its terms.

What we experience is a situation in which different levels of locality are enmeshed in a complex 'felt' of layers: local and regional traditions and cultures, national and international politics, international financial markets and transnational incorporations. Rather than by 'globalisation', our situation is characterised by translocality, in which different local agents, individuals and initiatives, operate within a networked environment. Translocal means that you are dealing with individual local situations but they are distributed within a larger geographical and cultural system. The global is locally embedded. In the best case, the global is a learning process. If we want to understand the global as something that we can work with, we have to understand ist forces and layers and also to understand how it is connected to the local.

The effects of the described situation are, in East/West Europe, exacerbated by the political changes in the former Eastern Bloc on the one hand, and the restructuring of Western Europe in the name of the European union which is still, more than anything, a European economic community.
The tensions arising from this remapping of Europe can also be felt in the arts sector, and we felt that we could tackle and diffuse some of the problems by connecting to others, asking questions, telling stories, exchanging ideas, and working together.

In the winter of 1995/96, V2_Organisation launched ist 'V2_East' initiative which has since been a conceptual attractor for different network, collaboration and presentation projects that were held in Rotterdam and elsewhere. V2_East is dedicated to enabling and enhancing contacts and co-operantions between people interested in media art and media culture in Europe. The most important result of the V2_East initiative has been the formation of the 'Syndicate' network. The name came from a comment that Vladimir Muzhesky from Kiev made during the initial V2_East meeting at the end of the Next 5 Minutes conference in Rotterdam in January 1996:

Individually, we are rather weak when it comes to negotiating with funding bodies and governments about support for new media and electronic art projects. However, if we could join up and form something like a syndicate, then we would be able to speak with one voice when it is strategically necessary, and become more powerful than we are now.

Since that first meeting of 30 participants, the Syndicate network has been growing steadily. It arose from an East-West co-operation idea and is now, in the late summer of 1997, a network that connects more than 170 people from 28 European and 3 non-European countries. The East-West axis is becoming less and less relevant for defining the character of the Syndicate which is, at the moment, turning into a European platform for media culture and art. We meet regularly in the context of festivals and conferences, like at the DEAF festival in Rotterdam (September 96), the Video Positive festival in Liverpool (LEAF, April 97), the Hybrid WorkSpace/dX in Kassel (Deep Europe, August 97), or at the ars electronica in Linz (Net.Shop, September 97). We organise publications, either by producing cheap readers or by editing special issues of magazines. There are always more ideas and possibilities than can actually be realised, but high on the current wishlist are a Syndicate Publication Series, the creation of a Translators Network, and the formation of a Student Fund for Digital Media Training.

The most important communication channel of the Syndicate is a mailing list through which we exchange information about upcoming events, about projects and ideas, and which we use to circulate texts that are interesting in relation to our work. A website is used for archiving messages from the list, for collecting Syndicate-related material, and for linking to the websites of the members of the network.

There is a continuous discussion about the different labels that we use: the 'East' dimension can be seen as a neo-colonialist effort on the part of Westerners; the 'Europe' dimension has become increasingly problematic at a time when the continent is as much of a cultural and political patchwork as ever, when war is waged about competing versions of history and identity, and when the continental powers are trying to build a fortress against the 'rest of the world' which, after centuries of exploitation, is bringing 'reality' back home to the Europeans. As Calin Dan put it: ''Etre EUROPEEN' is both as weird and as dull as being no matter what other kind of excessive minority. Because the EUROPEANS are a minority, and if the political class developed from this complex an irritated attitude on both home and global matters, we - the average EUROPEANS - should try to experience it without such a ridiculous intensity. More relativism concerning one's origins could do only good. And reversing the aggressive question from the other to the self should be an exercise of polite(ical correct)ness not only for the EUROPEANS but also for other fundamentalists.'

We therefore use the mentioned labels not as 'black holes of identification', but as heuristic tools for describing a temporary situation or for writing grant applications, and try to be as ironic and reflexive about them as possible. The spirit is pinpointed by Vuk Cosic's proposal for the Ljubljana Digital Media Lab to create the 'Ljudmila_West' initiative that would support Western media practitioners and help them to learn from the Eastern expertise.

For the Syndicate workshop at the Hybrid WorkSpace during the documenta X in Kassel we chose the title "Deep Europe". We were looking for a term that was neither East- nor West-specific, that carried some of the historical baggage of the notion of Europe, and that was at the sametime strange enough to be easily understood as ironic. It was an experimental title that turned out to be an interesting focus for thinking about the context of our work. In the end, Luchezar Boyadijev's (Sofia) reading of 'Deep Europe' was accepted by most participants: 'The notion is a metaphor which could be problematic. In the logic of this metaphor, deepness or depth is where there are a lot of overlapping identities of various people.

Overlapping in terms of claims over certain historical past, or certain events or certain historical figures or even territories in some cases. It could also be claims over language or alphabet, it could be anything. Europe is deepest, where there are a lot of overlapping identities.'

This mapping of culture and of the depth of identities onto the mental and physical geography stands not in contradiction to, but is a condition of the work that is being done in electronically networked translocal environments equipped with all sorts of telematic gear. After the workshop, Branka Milicic Davic wrote: 'what is deep europe? is it real? is it safe? my answer is - yes. deep europe is real. it exists. i do not need visa to be there, i do not need an invitation letter to be there, i can simply sit and think and i am there - in the land without borders, policemen, elections, president, government.... where no radio or TV station will be banned... whose citizens are speaking different languages without shame... and lot more. deep europe is my homeland, my private mental space, which i share with others. deep europe recognizes words like exchanging, sharing, growing. and that's why i believe deep europe exists. because i went there, and i can go there whenever i wish. to exchange, share, grow and understand.'

On the surface, the Syndicate is an informal network and an 'intercom' system for people in the media art community in Europe and beyond. At the same time, this inter-communication effects a re-mapping of cultural and mental territories that transcends the political, religious and territorial separations which we regard as a temporary nuisance, rather than as the last word on this imagined continent/container. Lisa Haskel (London) concludes her Deep European 'letter from home': 'So perhaps, this is what Deep Europe is all about. Not a political position, a utopia or a manifesto, but rather a digging, excavating, tunnelling process toward greater understanding and connection, but which fully recognises different starting points and possible directions: a collaborative process with a shared desire for making connection. There may be hold-ups and some frustrations, quite a bit of hard work is required, but we can perhaps be aided by some machinery. The result is a channel for exchange for use by both ourselves and others with common aims and interests.' At some point in the future we might reach a situation in which the heuristic tool V2_East - as was formulated from the very beginning of this initiative - will have made itself obsolete.

A "film director" is someone who, possessing a few of these disks, re-uses this material on their bookshelf or their terminal.

The communication sphere (in other words, the world) split in two in 2020: the public and private networks diverged from each other in such a way that while it is possible to make the transition from the public to the private sphere, moving the opposite way is now out of the question. These restrictions, in addition to the obligatory personal user codes, are enforced by a clever signal system similar to traffic signals. The authorities can also radically interfere with any illegal private-to-public pirate activity that they uncover (the so-called information explosion means that in a matter of seconds, information from the public sphere swamps the pirate station so that it becomes inoperable). While the public sphere is enmeshed in complicated legal regulations, the private is ruled by anarchy. People, apart from the small group mentioned below - also fall into two groups: the bigger film directors and the ever-decreasing audience. There is rarely a link between the two.

As practically every imaginable picture is now available using simulated processes assisted by cheap software packages, and can be substituted, supplemented or modified, the fashionable twentieth-century expression "shooting" has been modified in everyday speech: it now refers to retrieving images from the disk. Production companies aware of this are pushing their capacities to the limit and have introduced a new system of image assessment which expands the two-day shoot capacity to a gigantic forty-day shoot capacity - equivalent to approximately a year of television material. After a short (one- or two-second) object text or graphics motif sort, any part of the material is immediately, in real time, ready and modifiable (this means a 4,380 two-hour long films; the more famous directors are already equipped to work at this pace).

Of the most popular software applications, the destiny-matrix and destiny-catalyzer are worth mentioning. In the earlier destiny-matrix program, 256 destiny-patterns were available (256 basic situations, with 256 possible roles, representing a complete life span for each one) in a matrix-type hierachical arrangement, so that the various destinies could be combined with certain of their elements. The destiny-catalyzer, on the basis of selected material ("roles", characteristics, careers), randomly suggests film possibilities - in the case ofeight to ten roles (dependent variable) and however many extras, with whatever backdrop (independent variable), it generally suggests forty-five to fifty ideas. These can be immediately recorded, and if the director decides a particular scene is worth producing, he or she can immediately switch into the legal info-bank network with the help of an aid program and, in a matter of days, retrieve the desired elements. This is a very practical step, even before an outline of the film has been completed, because the cost of such elements changes daily. In fact, it is almost impossible to avoid taking this step if directors want to arrive at a final celluloid product (which of course can only come about with the proper state protocol procedures). The info-bank immediately links up with the original compostion elements in its database, and with the archive or national database which stores the original rights, in order to obtain the desired elements. At the same time, the info-bank arranges a proposed completion date (completion often takes years).

Most of the software writers (destiny constructors) have hundreds of programs (not always compatible) on the market (Destiny Trailer, Magic Destiny, DesTiny for the young etc). The most popular is Microsoft's successor, the Macrohard Doors package (on its cover Aristophanes dances with Greta Garbo in front of Ludwig of Bavaria's castle on the ice of the Nile). Its name derives from the introductory screen, which opens doors into the lives availbale in the virtual castle.

Just as the destiner-catalyzer can automatically put together fifty or more film sketches if the destiny-matrix data is given correctly, the cheaper B and C film directors often do not bother to make more than one film: they simply release several versions of the same thing, arranged in a series or masked in genre categories. For example, a fictional B film about the love between Nijinsky and Mata Hari, created from illegally obtained archive material, can be seen in the "social drama" Kiss of the Cheka, as well as in sci-fi, horor, soft and hard porn, and even animated versions, depending on the atmosphere and the age of the viewer.

Archetypal directorial works are released in "skintight" format, which allows every consumer, with the help of their own recorder, to take part in them. Viewers can participate directly in the three- and four-dimesional scenes. (This is not advisable for weak nerves, especially because of the cuts, although in the more fashionable universities the viewing of these versions are obligatory in the first year.) Here the viewer is part of the film. They can imagine themselves playing a role in any scene. They do not look at the film but look around in it - they experience it. (With the commercial appearance of these devices, the term "empathy" has disappeared from the psychological dictionaries and has been replaced with the expression "act out", which in essence refers to the new mimesis practised by inward-looking personalities. I will return to this later.) This film production form is popular in blind circles, since the time-freeze allows them to virtually feel their way around scenes. (The cure for blindness itself has not yet been found - not everything can be solved - but all things considered, this is not such a handicap.)

In this "world", everybody can be virtually (visually, aurally, and spatially) taken into any situation. The law divides people (both living and once living) into three categories: 1) general, 2) film actor, and 3) outlaw (criminal). The right to visualize other people's lives is regulated in the penal code on the basis of these categories. Only the images of those who fall into the third category can be freely used by anyone (although there are also certain limits), while those in the secondcategory "can only play in an exclusive context in which they have already played", and only with an agent's permission (which is naturally the source of numerous legal debates). The material of those in category 1) can only be used by a small circle with family permission and if this cannot be obtained then not at all. (Legal protection of the individual is extended to the human image, too.) Permission for one-time use can be obtained whatever the case and often at a pretty price - amounting to several times the price of the whole film. Screenings without this permission, even for public performances, are illegal. That is why big agencies often ask for permission months in advance.

Film makers usually choose the fourth option: employing figures produced with simulation techniques (every cheap software program offers a few). The advantage is that the simulated image is capable of really spectacular rapid metamorphoses, from fairy-tale witch to any form desired. The first work using soley this type of image was a popular remake of Ovid's Metamorphoses, and this type then became known as the metamorfoid, as opposed to the android.

The one problem with metamorfoids is that both copyright law and civil law prohibit the "metamorfoid use of motifs resembling any once-existing destiny or person". The developers are often forced to withdraw certain elements or metamorfoids from the software market because a real person intentionally decides to live just that destiny - "new mimesis" is the proper term for this well-known trick - and when decisive elements in that person's life begin to resemble the metamorfoid's, then they demand compensation. Generally it is more economical to put a new program package on the market and withdraw the old one than pay the high costs involved.

For all intents and purposes, acting has become the only form of military service, since the traditional, historical form, with its ranks and long training periods, has become redundant. The already existing acting material is easier and cheaper to use. Military developments naturally have not ceased, indeed they turn increasingly to the fields covered by the biofilm program. The purpose of this research is to program any type of drama directly into living organisms, through the eyes, through sight. On the consumer side, it has been shown that a viewer can watch 178,850 films continuously, on the basis of a fourteen-hour a day viewing day, two-hour films, and aseventy-year viewing career (excluding childhood, when age restrictions apply). Film viewing in this form is not merely the twenty-first-century equivalent of bodybuilding (many people wishing to get into the Guinness Book of Records have registered themselves for the 200,000-film "dream limit"). Film viewing has also become a vital source of subsistence. The radical drop in the normal number of viewers (at the expense of the film makers) has brought into existence a new career structure. The state, and indeed local government, now deals with the viewer as a paid public employee, so that certain poorer families sacrifice one member to this unpleasant work, in order to rely on a permanent income. The danger in this - the new socialists call it "infoslavery" - is that constant viewing is like the strongest of drugs, and those who take this work can never, after long exposure, be cured of it. The post-information society needs this public-service sector, as it gives significant financial aid to the moving picture industry on the basis of viewing figures. Given the number of filmmakers, however, the revenue generated, when broken down for one person, does not even guarantee minimum subsistence. Thus, there is a serious battle to attract viewers; and programs for reliable viewers in good health are reserved years in advance (often using montage materials for productions that are not even ready).

Only directors with complete life-work plans have a chance of winning the tenders - which are essential to the maintenance of the public communication sphere - often adding press reviews and monographs on their oeuvre about films they have not yet made.

Most of the material does not even emerge in final form as it gets used up and disappears in the networks and the interactive personal TV (popular in the private sphere, IPTEL for short). The quantity of transmissions is so great that on general holidays the interference clouds can interrupt the weather, and a central infoblock similar to a smog alert takes place. At these times, everybody can only communicate by traditional means until the end of the alert. (This system is subject to abuse by the big advertising agencies, who have power over the weather office.)

Statistics of information highway accidents is dispiriting. The most frequent cause of life-threatening accidents is the otaku virus (a form of it also appeared in the last decades of the twentieth century): If someone tries to enter a completely disinterested field and gather all the information belonging to it, they can cause a series of infocrashes and provoke so-called info-harakiri on a local level, as well as spread several hundred known incurable viruses.

Even the church cannot avoid highway building: Their satellite, called Man of Light, permanently transmits a rainbow which can be tuned into, either directly or via the networks. Indeed, the scientific societies, which are among the poorest type and whose numbers are rapidly diminishing, also operate a Quack, Quack network. A peculiarity of the latter is that it merely transmits newly received information - it does not store anything or make replies.

Although popular opposition to this all-pervasive system of networks is seemingly very strong, very few people (as the above examples show) are capable of truly remaining outside it. Four such groups can be named: a few deeply religious families; a sect of believers in a natural way of life which denies the image as a natural phenomemon; a very small group, perhaps less than a thousand, of the very rich; and the radical artists. Given that they cannot join the highway even in the slow lane, their remaining outside means that nobody knows about them.

In contrast to the above groups, there are also exhibitionists and families who wear miniature cameras on various parts of their bodies which they keep continually at the ready, armed with permission from the personal TV networks, and at the disposal of the satellites (EXISTEL).

A similar asscociation in the public sphere is the documentary makers society. Its delegates wear cameras on their foreheads - as a third eye - with the aid of a specially made Doc-Hat. Through this device, the cameras are switched on and off in the central documentary office. Approximately five minutes before being switched on, a red light shows and a whistle sounds from the hat, giving anyone who does not want to grant image permission the chance to get away.

The avant-gardists have revived Jean-Luc Godard's dictum - "Every second of a film is twenty-four frames of truth" - and Malevich's black square, and have organized themselves into a cartel. They are only willing to work within a prescribed form: a strictly controlled playback speed (twenty-four black, upended frames per second), with a truth inscribed in every frame. It is no wonder that the anonymous author of a children's film about animals made in the year 2095 put into the mouth of a now-extinct crested newt the sigh: MEHR LUMIERE!

Invited one by one into an anechoic chamber, a visitor's first sensation of Mikami's piece is likely to be fear, fear of being locked in. There is an extraordinary sense of claustrophobia rarely encountered anywhere else. One is cut off from the external environment both visually and acoustically--such is the initiation to the whole event. With no environmental sound and no echo--the ears feel clogged. The room is small (5 square meters) like the inside of a cell, prison or biological, with walls completely covered by a fimbria like material--geometrically shaped yet oddly organic, wildly indented, and of skin color. Waiting inside is a chair obviously meant for clinical use.

As soon as the connection with the external world is cut off, one's attention shifts inward. Gradually we begin to hear our own internal sounds, the sounds that we all make, inexorably, as long as we live: various growlings of the stomach and intestine, the gurgling of fluids flowing everywhere, surprisingly big rustlings of, perhaps, dried ear wax, and finally, the most dominant--the repetitious yet irregular pulse of the heartbeat. The orchestration of this amazingly rich range of sounds comes from somewhere strange--so close yet so far. There is a sensation of being suspended in the bottomless pit of a biological sound/environment; this is the music of the abyss, the groundlessness of one's own body.

By definition, anechoic (echoless) is different from soundless (which occurs in a vacuum, for example) and, of course, silence, that is a rhetorical rather than physical situation. Even in a busy urban space, silence can arrive whenever we pause to notice it, when we stop our conversations with others, or shift our attention from the normal routine. Our environment consists of layers of sound in which silence is a momentary act of shutting off certain layers in order to pay attention to the ambient. It is a technique of shifting our attention by what phenomenologists call subjective bracketing. Therefore, it is possible that, even while listening to street sounds, we can achieve silence in our consciousness by transposing whatever noise there is to the realm of inattention. In this sense, the production of silence, which is mainly concerned with dialogic structure or communication, is a cultural event par excellence. But being caged in an anechoic room deprives one of the context within which to perform such bracketing. One is thrown into a situation where the ground is decomposed, and a solitude that is abysmally rich opens its mouth. It is said that dogs and other animals can live only for a short while in anechoic rooms because of the disorientation, and there are records of anechoic rooms being used for torture.

This is the prelude to the main event. After being invited to sit on the chair and attach a stethoscopic device near the heart, one hears and sees an edifice built of one's internal sound/environment. This is the moment when the abyss organizes itself into a variety of spatial configurations. Having been compelled to oneself become a 'gigantic ear' of the abyss, attention now shifts to actively producing various forms and sound patterns. First, the repetitious sound--ba-bumph, ba-bumph, ba-bumph, ba-bumph--of one's own heart begins to come from outside instead of inside. Then, the room is darkened, to the surprise, fear, and perhaps expectation of the audience. Curiously, my own heartbeat comes from somewhere behind my ear, in the beginning from far away and then gradually near, then from everywhere at the same time. It comes and goes from various spots and toward various directions. Sometimes it sounds soft like a stream and sometimes shockingly aggressive like a gun shot. All of a sudden, it begins to rotate around my head, and it scatters all over the space. Then, along with the roving heartbeat, my own being feels like moving around the room and forming various spatial patterns, accompanied by my pulse as the breath of the world itself. Becoming sound, 'I' constructs a space/time architecture of another dimension. 'I' is totally dislocated, inside-out, losing the solid locus that keeps the daily identity. I(t) feels like becoming the space of the room itself--the gigantic ear. After eight minutes of this transformative spatio-temporal construction, the lights come back on and again the ba-bumph, ba-bumph lasts for a minute, as in the beginning. The denouément: one is escorted out of the room. A total of 15 minutes by the clock, but an immeasurable time that feels endlessly long and instantaneously short.

Less overwhelming but equally important is the visual component. During the eight minutes of darkness, a growing, transformative abstract net pattern is projected on the fimbrial wall. Looking at the biomorphic growth, it gradually becomes apparent that its movement is in synchronicity with the acoustic spatial pattern. The acoustic experience is being visualized into a transformative net scheme, and watching, one comes to be able to detect and measure the spatial form of one's own soundmaking and hearing experience. It might be said that because of the visual reference, one's consciousness manages to keep its integrity. By offering the spatial orientation, the visual can hook and sustain the body/soul integrity that is on the verge of falling apart. The subject, first thrown into the abyss, is reconstructed in the spatio-temporal architecture that autonomously grows in tandem with the visitor's biological condition.

Case history

For the past several years, Mikami has been asking audiences to lend their bodies to her work. In "Borderless Under the Skin", for example, the throb of our own pulse was borrowed, and in "Molecular Informatics", our gaze: in all of them, the data of our body was the program's interface. In "Molecular Informatics," the audience, which was invited "to see" the work, donates its own gaze to the program, where, as participants, they directly encounter the split moment between their existential passivity (reactive eye movement) and cognitive activity (the intention to see), always already existing at the core of their subjective formation. But the role of seeing in the current work, "World, Membrane, and the Dismembered Body" is the reverse, that is, instead of inducing the moment of split, "seeing" delivers a certain recovery of the split by way of construction. The differences, connections, and developments between the two projects seem to correspond to the nature of the hearing and seeing circuits in our bodies. According to Mikami, the function of the ear/hearing is more passive than that of the eye/gaze. "Unlike the eye, the ear does not have a double-fold, self-conscious expression. The ear does not express that it is hearing. The ear cannot move at the owner's will. While the eye can be shut off to intercept certain incoming information, the ear is totally passive. It simply receives, without being selective. The ear does not seek, either. Sound offers only its orientation and movement to our acoustic perceptor, the ear; we cannot easily detect the sound source solely by hearing if not for the support of visual perception."

"Molecular Informatics" and "World, Membrane, and the Dismembered Body" deal with the two major sensory data--visual and acoustic perceptions--that today's cultural institutions rely on most. These are the domains where today's culture most powerfully and actively produces phantasmagoric illusionism. Mikami's manifesto for her two projects is that "the ear is not a thing that merely hears, the eye is not a thing that merely sees." This offers an important clue to understanding their common denominator. They both address the discrepancy between the function of the fragmented sense-organs in our world of the techno-cultural network, and the way the cultura-lingual symbolic order customarily rhetoricizes it. We are the object of capital's agenda to sell the information commodity at the same time as being the subject who judges and buys it. At the same time, our Being or Subject is fragmented into the beings or subjects of different sensory organs: the thing that sees (i.e., ads, films, TV, art and so on), and the thing that hears (i.e., prayer, agitation, music and so on). Mikami's work makes us aware of the passivity of existence by emphatically decomposing an authentic human being--the body/soul integrity--into functional parts already implicated and inscribed within the network of the invisible whole we tentatively call the 'world.' Our subject thinks/says that "I see", "I hear," while it is also made to see and hear in the context of technology and scientific analysis. In this way, these works create an arena that hosts breathtaking struggles between our existential passivity and activity. The struggles present the way the 'world' is being constructed everywhere, day by day. In this sense, Mikami's work presents an anatomy of the contemporary Cogito, the thing that sees/hears but not always doubts.

What is at stake in "World, Membrane, and the Dismembered Body" (hereafter shortened to W.M.D) in particular is the passivity of acoustic perception, even more intense than that of visual perception. Mikami's statement, that the " pulse of the heart beat is the most base of self expressions," alludes to the limits of cultural production--from which it appears and to which it disappears--the status of our existence, simply being here, is already an "expression." As the work makes us realize so vividly, we have always been an instrument, even before listening to any sound. If our heart is not beating and our eardrums are not vibrating along with it, we cannot enjoy sound/music from the beginning.

The heart/ear circuit is extracted from the daily environment (or the body/soul integrity) and recomposed in a new context in a way that the Subject is toppled from the Imperial throne from which it observed, scrutinized, defined, and dominated the object of perception (as an audience of music or cultural products in general). In a situation when the sound source and the sound perceiver--or artist/player and audience--are not safely distanced but collapsed into a mutual nesting box, the subject of the audience is made to face a critical chasm that, in fact, always exists at its genesis. It so happens that, as Gilles Deleuze said, "'I think' and 'I am' do not correspond. The chasm is the motor of time." The chasm is the cruel moment at which the split of the Subject attacks, yet it is also the moment at which a new spatio-temporal production begins. At this instant, the pulse of the heartbeat strikes the hearing subject as a preemptive attack. Without any preparation, the Cogito is attacked by an accident, the accident of one's being in the world. Listening to one's being in W.M.D is equal to listening to the accident of the world or the world as an accident.

What "World, Membrane and the Disappearing Body" presents to us is that the heartbeat is the sound of sounds, or the music of musics. One might even say that the heartbeat subsumes all virtual sounds in the same way that, in the psychoanalytic sense, the phallus subsumes all virtual objects. But in W.M.D, what is at stake is not the heart per se but the hEARt circuit. Like a Klein bottle, a stream which comes from the heart is absorbed into the gigantic mouth of the ear, which is again connected to the heart. The "ear" interface, the place of exchange, stands on the meta-level to the unconscious, contingent "heart," at the same time they are collapsed into one continuity. In this scheme, it is not just a natural phenomenon of body that is reconstructed, but a moment of production--the human machine that is connected to other machines. In the whole of the productive machine, "the head [ear] is the organ of exchange, but the heart is the amorous organ of repetition." (Deleuze)

With respect to the audience's experience of the work, Mikami observes: "It is a real time, double-fold, self-referential status: the sounds the audience's own bodies make--subtle internal sounds made by organs reverberating between tissues and membranes--are proliferated outside the body via the device and further reverberate in the room to affect the ear of the audience, which itself is a part of sound sources. In this work, nothing--neither body nor the device nor the environment--is the main objective to be experienced, except that the 'ear' stands out as a mediator between systems, expressing itself as a certain perceptive coding of topos." W.M.D is this "topos" rather than an image, for representation of a certain object--image, view, narrative, thematic system--is not the concern of the work. Rather, it produces and reveals the mechanism of representation itself--how the representing subject and the perceiving subject are part of the techno-cultural productive "machine." This points to the view of the world not as a "theater," but as a "factory" (Deleuze/Guattari): the world as a gigantic factory in which representations (theaters) of the world themselves are a part of the production.

Diagnosis

In Mikami"s work everything is arranged around the body or the body senses. She produces work which employs "fragments of the body (sensory organs) by decomposing them into data" because she believes that "essential elements which compose interface for programs all exist inside our body." Then, what is the "body" the audience owns as well as experiences in W.M.D?

Paul Valéry once categorized the four kinds of body we have in modernity. The first one is the "body for the self." Valéry stipulates: "Each of us calls this body My Body; but we give it no name in ourselves, that is to say, in it. We speak of it to others as of a thing that belongs to us; but for us it is not entirely a thing; and it belongs to us a little less than we belong to it. . ." In W.M.D, what we experience so vividly is the very ambiguous mutual belongings of our subject and our body--the abyss. Valéry continues: "we can say that the world is based on it and exists in reference to it; or just as accurately, with a simple change in the adjustment of our intellectual vision, that the selfsame body is only an infinitely negligible, unstable event in the world." Or even more appropos to our problematic concerns, our body is the primary obstacle to our ideals. The resistance of our body to our thoughts is the primary sign of the trauma that the world does not belong to us; it is the alterity. To our Cogito, this life, this being, is nothing but an accident which we nevertheless have to believe to be self-affirming or even causa sui. The world of passion (passivity) begins from and is based upon our bodies for ourselves. To the Cogito, the social relations, including brutal conflicts, and our bodies, including illness, connect to form one and the same world of wars. The body of the first category, Valéry states, is "formless"; "it has no past." This amorphous present of the body is the premise of Mikami's work.

The second category of body we own is "the body for others." This is the one that is looked at and judged by others, the body that is thrown into the social world. For us to see, we have first to be seen by the other. For Jean Paul Sartre, Maurice Merleau-Ponty, and Jacques Lacan, being gazed at by the other is the initiation of the subjective formation. The body touched by the other also offers the paradigm of love. Furthermore, this particular circuit produces the cultural code of narcissism. In this work, we are the other.

The third category is the body that appears as the object of scientific observation in the context of modernization. It is fragmented into objects of analysis, operation, and manipulation. Heidegger as well as Valéry had reservations about the fragmented and instrumentalized body--as a loss of Being or the wholeness of nature/spirit. In fact, all the fragmentations were concurrent. Thus the problematic of technology or techné in the modern is provoked by the advent of the third body. Mikami's maneuver is rather an affirmation of the fragmentations; it pushes them to their limits. The elaboration of her technology is a critique of or competition with the seduction of the "authenticity" that Heidegger evokes.

Finally, the fourth body is the most tricky. Using an untranslatable term, implexe, Valéry indicates the body that is both real and imaginary. In today's terminology, this is the virtual body: the totality of the body that we own/experience yet can never grasp at this or that moment, by this or that system of thought. This is the complex of all bodies we own/experience as a being-in-the world. According to the philosopher Hiroshi Ichikawa, the body as a realistic synthesis is possible only thanks to the potential of the implexe, though it is realized, in a sense, only by deterring and going beyond the implexe. For this reason, the body as a realistic synthesis is deemed both revealing and veiling. Upon the accomplishment of the synthesis, however, the implexe is buried under the subconscious. This fourth body--inasmuch as it is neither a realistically synthesized nor revealed object--is beyond phenomenological description.

W.M.D reveals and produces the topos wherein the four categories of body all intervene. The audience faces the first body as an abyss; s/he experiences a moment very close to narcissism when they enjoy their second body as if it is another's; the third body is engaged by the artist's sophisticated programming; and last, what we ultimately experience in the small room is the moment at which this indescribable thing--the implexe--constructs the spatio-temporal edifice, namely, the extension of body.

Employing the implexe, the invisible drive, this project constructs a unique environment by making the audience's internal environment external. This "inside-out" by implexe might be the essence of the architectonic. The way our body is both proprioceptive--receptive to it's self-produced stimuli--and exteroceptive--responsive to outside stimuli--or how it is autopoietic and allopoietic, can finally express itself. Perhaps human production of its own environment--what can be called Architecture--is always a way of externalizing the internal environment. Virtual space, the fourth body, the implexe, assumes the potential for structuring the "inside-out." If not for the inside-out, there wouldn't be any "here and now, or rather an Erewhon from which emerge inexhaustibly ever new, differently distributed 'heres' and 'nows.'" (Deleuze) Just as Samuel Butler's anagram "here and now" is essentially produced by "nowhere," the productive "nowhere" can become "now here" only by way of the jump of inside-out organized by the virtual body.

Operation

Mikami's work is not an art that expresses something, taking for granted the authenticity of art or the world of representation. It is not an art that employs technology and media for a certain effect of representation. It points to a situation where art and technology are not yet separated, and stresses the precedence of technology. As we have seen, W.M.D creates the circuit through which our body extends itself in the techno-cultural network by an elaborate censor of perception. For this mechanism, even a description of audience participatory/interactive art is not sufficient.
Rather it is an art that constructs a particular way we make the world our own body and live in it--the productive force of technology. On another level, Mikami's work intensely and even cruelly presents the way we are fragmented and becoming part of a world that is constantly expanding in tandem with technology's brutal, accidental force. Making us directly experience the ambiguity of technology, Mikami's work presents the immanence. Yet it refuses to represent the whole world in which we are apart. Mikami composes interfaces of various body parts and connects them into a growing network which does not presuppose any existence of a whole. It is a production of different machinic arrangements. In this sense, it is engaged in an affirmation of chaos. Since the work itself radically persists in being ambiguous--an interface--a conclusion is hardly possible. But as a final thought without a final definition or a last instance, I would say that Mikami's work is a mediation between two philosophical plate-tectonics that have been shaking us for a very long time: those of Being and chaos. Which is to say that it is persistently a mediation and an interface which refuses any finality. Futhermore, being a patient, a guinea pig, an organ donor, and an audience in her work, I confess that I became totally dissected, disoriented, and lost in the sense of observing and speaking about it as an art work. Being in Mikami's work does not allow the audience to be a safe, unaffected observer, but forces us to begin to confront the work in the midst of a total breakdown and resurrection.

References: Gilles Deleuze, Difference and Repetition, trans. by Paul Patton, Columbia University Press, 1994; Hiroshi Ichikawa, Seishin Toshiteno Shintai, Kodansha, 1992; Paul Valery, "Some Simple Reflections of Body", included in Aesthetics, trans.by Ralph Manheim, Pantheon Books, 1964.