Introduction
Shuffles and Seeds confronts people with the temporality of our existence. It questions what leads to self-identification and identification as a human. Shuffles and Seeds is a single-player VR experience. We introduce the player to an embodiment, an avatar, in Virtual Reality that goes through different cycles of construction and deconstruction. As the player moves in VR they create memory traces that are replayed on a separate avatar with which the player can interact.
After some time that avatar is deconstructed and dissolves again within the environment, only to be recreated in a new cycle. Across multiple cycles, the avatar evolves from just a copy of the player — to a blend of multiple memories created by the same player. In the next step, we also explored the boundaries between self and collective as we base the avatar on the memories of multiple previous players.
Shuffles and Seeds was funded and supported by Creative Funds NL, with their Experiment call
Goal
ImproVive has always been interested in exploring embodiment and identity in VR. With Shuffles and Seeds, our goal is to explore how people perceive identity as the boundaries of an avatar’s embodiment blur during the process of construction and deconstruction. We also want to explore how much can still identify with an avatar as we blend multiple memories of your previous movements over time, when does your feeling of identification change. What happens when the memories of multiple people, at what point do you disconnect?
What defines us, and where lie the blurry lines of self? What kind of influence do we leave in our world as we move through it? What is left of your impression when parts are reconstructed from earlier memories, or even recreated from a shared history with others? Can we make algorithms that mimic, learn, and attempt to deconstruct and construct those concepts?
The underlying components of Shuffles and Seeds
Construction and Deconstruction
We humans are machines to carry our genes, constructed and deconstructed momentarily in time. While we perish and get created over and over again, the code that constructs us, lives forever.
We wanted to explore construction and deconstruction from this lens. We realized in our early tests that approaching this via VFX, or Particle effects in a game engine is difficult. Since it was the end-of-the-line rendering method and not a self-organizing particle. We came up with the idea of autonomous particles. Can particles organize themselves and make up a form and follow a form?
We looked at mathematical patterns that exist in nature. One of them is the Collatz conjecture. It is a simple pattern based on random seeds, and following the simple rule of Collatz’s conjecture.
What is Collatz Conjecture?
The Collatz conjecture is one of the most famous unsolved problems in mathematics. The conjecture asks whether repeating two simple arithmetic operations will eventually transform every positive integer into 1. It concerns sequences of integers in which each term is obtained from the previous term as follows: if the previous term is even, the next term is one-half of the previous term. If the previous term is odd, the next term is 3 times the previous term plus 1. The conjecture is that these sequences always reach 1, no matter which positive integer is chosen to start the sequence. The conjecture has been shown to hold for all positive integers up to 2.95×1020, but no general proof has been found.
We observed that the pattern was interesting, but it was fairly unpredictable and often grew much larger than the general outlines we recognize for embodiment. It is a principle we find interesting for other experiments but we decided to not use it for Shuffles and Seeds.
What we did do is create a collection of particles that follow the outer shell of a human avatar. These particles are distinct and follow their own dynamics. So they are created and disappear, and can be influenced by touching other objects and also can be blown away by the wind. The idea is that the player feels that their form is not constant but constructed.
Over time the player is confronted with a cycle of construction and deconstruction. During construction, the particles grow and fill the hull slowly defining the body. During deconstruction all the particles dissipate or fall away, taking away any reference point to their own body for the player.
The particles also detached and fell when the player interacted with the environment or the other player.
Introducing a second avatar — copy vs mirror
In our past projects, we have often introduced a mirror to provide the player with a reference for their embodiment. In VR it is difficult to see your own avatar without the use of a mirror, just as in real life you cannot see all of your body without a mirror. In Shuffles in Seeds we wanted to have something more than a mirror, a 3-dimensional copy of yourself as a separate entity. We wanted a body that occupies 3d space, so the player can interact with it.
The most important reason to also choose a copy is that we wanted to play with the fact that it sometimes mimics the player, but sometimes shows another animation.
The copy is just as the player constructed from particles, but the particles themselves are not copies but interact with objects independently. The copy follows the same cycle of construction and deconstruction as the player.
Environment
The environment provides you with a reference point. We didn’t want to leave the player in a void at the moment of deconstruction. So we chose to have some of the environment static. At the same time, we wanted the player to be able to influence the environment, and be influenced. So within the reach of the player we created a number of objects that cycled through a construction and deconstruction phase just like the player. Through touch, the player could also interact and influence and be influenced by these objects.
Body Types and mutation
In our construction and deconstruction cycle, a player might lose interest to explore, if the body was exactly the same every iteration. So we explored an evolution of body types with each cycle of construction and deconstruction.
We tried two different mutations adding wings to the embodiment and adding octopus arms and legs. We would cycle through different mutations at the start of each construction cycle.
Growing wings
At first, we tried growing the wings over multiple cycles, but speeding it up to complete within one cycle felt better. It was also added to the copied avatar. Autonomous particles attach to the parts of the wings. The wings followed their own animation. They looked nice, but the player had no real control over their movement. Also, as they were on the back of the player’s avatar, you could not really see your own wings, only the wings on the copy.
Octopus Arms
A second mutation we tried was adding octopus arms. Just like real octopus arms, the arms move on their own. But here we decided to put them not on your avatar’s shoulder but replace part of the arm. We tried both the elbow joint and wrist, and the wrist offered the most satisfying result. We also added them to the legs, replacing the lower leg from the knee down.
The idea of the octopus arms came from the book ‘Other Minds’ by Peter Godfrey-Smith. This quote is a good example, “For an octopus, its arms are partly self”. He describes more about the autonomous nature of Octopus arms in his book.
Memories and Blending
We created a system to store the animation, select an animation and then replay it. At intervals we are recording the player’s movements during a certain period and store this in the database. Then we sample the player’s movement and compare it to earlier movements. In the picture, you can see such a selection of animations (the blue and purple avatars). The blue is the best match, the purple the other out of the top 5. The match is decided based on posture and movement. However, only the start of the animation needs to match what the player just did. The way it continues can differ.
This best-matching animation is mixed with the player’s current movements to create a blended animation. In the images below this blend between the best match (blue avatar) and the player’s movement (yellow avatar) is visualized by the brown avatar. In the pictures, you see a yellow avatar turn from yellow to brown as it mixes the two animations to create a new animation.
Images showing different matching animations and the creation of a blended animation (brown avatar) from a combination of the movements by the blue and yellow avatar.
In the experience, we don’t have all these different versions, but only the end result is used to project on the copy.
During the experience, the avatar copy is constantly moving through a cycle of alternating animations: from copying you to mixing your movements with a memory.
We discovered that it is important to set the balance right between copy and memory. Not enough memory in the mix and it feels like a sluggish copy of yourself. Not enough of the player’s own movement, and the player dissociates as it has no influence on the copy. Setting the mix to 50–70% of memory and 50–30% of player movement made it interesting. This is in contrast with our earlier experience with blending and control with Strings Attached. In strings attached, it was a two-player experience, where one player played in VR, and another player could influence the movement of the arms with a controller, and puppeteer the player partially. Since it was a first-person view, we needed to let the player have at least 40% control to make it interesting.
Creating a collective memory
Every time the experience samples your movement it is also creating a new memory trace. This is done multiple times during a play session. During a play session, you can choose to interact with only your own memory traces. At the end of a session, we can then upload these memory traces to a Collective memory (a database). We are also able to download all these memories. In that case, your movements will be compared to the collective memory, so all the memory traces are used to find the best match. In that way, we are sampling from a collective memory and the blended replay might contain your memory or that of another player.
Reflections
Deconstructing Embodiment
So far we have seen that embodiments that respond to the control of the player can feel human. We can anthropomorphize and project a lot of humanness even if our shape is defined by a collection of particles. However, we notice that if the particles break the underlying form and extend far beyond the shape and structure of a human form, it no longer feels like you. In our early experiments with particles, when we tried forming it using patterns of logic to project embodiment, we found there is a threshold where you can no longer identify it as a human-like entity. Even when the avatar follows the movement mapped to a human, it is hard to recognize it as a human. When there is an underlying human form, even something as strange as wings or octopus arms can still feel human. We did notice however that empowering interaction can make an embodiment much more powerful. An example are the wings. Their movement is not controlled by the player. It makes them feel less powerful, and less identifiable than the octopus arms.
As we cycle from construction to deconstruction there is an interesting observation: People really don’t like the final stages of deconstruction. Even though our embodiment is virtual, it empowers us to shape or space, to interact, to create new memories. All that is taken away as soon as deconstruction reaches it’s final phases. Our players would prefer to skip this stage. Most people find it difficult to be patient and experienced during those moments. Some do appreciate the cycles. But taking away not just interactivity, but all sense of embodiment is actually a quite profound experience
Self or Other?
Because we used real captures of movement from other players, the entity could take the appearance of another person. Even though it is a single player experience, you could feel like you are interacting with another being. We saw that adjusting the timing, and intensity of the blend was key in this particular aspect. You needed time to identify the new entity, And it was nice to see how you could still influence the other, and at the same time based on the memory it would do something slightly different. This nuance of difference was the balance between self and other.
As mentioned, your own movement from the same session is easy to recognize, but it is genuinely difficult to know if the movement is from a past self or from someone else. Since we had different bodies with added movement like the arms and the wings, if an earlier memory of a movement was projected on a different body it was more difficult to recognize that on the mutated body. In exploring the movements of the other it is very interesting to see where the copy plays along or deviates. During one of our tests a player started to mimic the copy, so the boundaries on self and memories blur creating a collective meme. We see that the new embodiments can invite players to participate in certain memes or interactions. A good example are the octopus arms. They invite the player to explore touching both their own, but especially the copied arms. This creates a common memory in the collective memory.
A note on a simplistic genetic algorithm of the autonomous particles
There are two properties we explored with the Autonomous particles. The stickiness factor and the lifetime. The particles started with the same values, but with every cycles of re-generation, they could become more or less sticky, or have more or less lifetime. The changed values could accumulate if it always added more stickiness or lifetime.
The stickiness time defined how long the particles stayed attached to the body, while the lifetime defined how long the particles existed in the current lifecycle. The particles were always reorganized and remembered these values in the cycles of construction and deconstruction.
Exploring Shuffles and Seeds with others
Reality check
We presented Shuffles and Seeds at Reality Check Festival in Enschede at Saxion University of Applied Sciences. Participants were also willing to add their movements to the collective memory.
Conclusion
Some of the ideas we explored, we would like to deepen and expand in future projects and experiments. We could explore a new dimension of embodiment and movement. We hope to add more movement memories to the collection over time.
Here are links to our other projects on embodiment in VR
Read our other blogs:
Questions? Do you want to talk more about embodiment, temporality, or VR?
Contact us at https://improvive.com/contact/
