Volumetric capture for narrative films

Key Question Being Explored in This Insight:

Krzysztof Pietroszek provides an overview of volumetric capture technology and its application to narrative filmmaking.

View the page of: Krzysztof Pietroszek

Explore more insights in:

Computer Science Formal Sciences Sciences

Media Literacy Toolkit

Faculti's Media Literacy Toolkit helps viewers critically engage with academic insights by analyzing the research context, identifying perspectives, and encouraging thoughtful evaluation.

Critical Questions to Consider ▼

What assumptions does the research make?
Are there alternative perspectives not explored?
What are the limitations of the research method?

Bias and Perspective Awareness ▼

This research comes from American University . Reflect on how the institution's academic focus and research partnerships may shape the questions being explored.

Further Reading ▼

Read the Study

Krzysztof Pietroszek provides an overview of volumetric capture technology and its application to narrative filmmaking.

Image courtesy of interviewee. March 19, 2024

Log-in or Sign-up to Faculti
Currently viewing this subject insight as a guest. You have insight(s) remaining for this month. Login to view 8000+ figures on the platform.

Generating Workspace...

Export as PDF

This video video is a comprehensive discussion about the speaker's interest in science fiction and its integration into their work, particularly in filmmaking and technology. The speaker recounts their journey from studying traditional filmmaking to exploring volumetric filmmaking—a method that captures three-dimensional images to create immersive storytelling experiences.

They emphasize the potential of virtual reality (VR) and holographic technology to enhance communication and storytelling by allowing individuals to engage with narratives as active participants rather than passive observers. Instead of traditional techniques, like montage or framing, volumetric storytelling offers a personal and immersive experience where viewers can explore the narrative space and interact with characters.

The speaker discusses a specific project, "Vera," which utilizes volumetric filmmaking to create an interactive experience where users can embody a character and interact within the story. They highlight the differences between this medium and traditional cinema, noting that volumetric storytelling represents the birth of a new medium that may coexist with, rather than replace, traditional filmmaking.

Furthermore, the text outlines the technological advancements and the democratization of filmmaking tools, making them more accessible through AI and improved equipment, leading to a new era of immersive storytelling. The speaker expresses excitement for the future of storytelling, particularly as technology evolves to enhance artistic expression and user engagement in storytelling experiences.

Language:

Export as PDF

Export as PDF
Here are five key points from the video:

1. **Integration of Science Fiction and Technology**: The speaker discusses how their fascination with science fiction, particularly concepts from works like "Star Trek" and Neil Stevenson's "Diamond Age," inspired them to explore innovative technologies in their work. This involves ideas of immersive communication and the development of virtual environments.

2. **Volumetric Capture Technology**: A major focus is on the potential of volumetric capture technology, which allows for true 3D recordings of people, objects, and environments. Unlike traditional 3D films, which are often two-dimensional films adjusted for 3D viewing, volumetric capture enables viewers to interact with these recordings in a more lifelike, immersive way.

3. **The Future of Communication**: The speaker envisions a future where traditional forms of communication, such as video calls, are replaced by immersive experiences that allow individuals to interact as holograms in virtual environments. This shift is expected to enhance communication by including body language and spatial convideo that are often lost in traditional methods.

4. **Immersive Storytelling Experience**: The video highlights a distinction between traditional filmmaking and volumetric filmmaking. The latter provides an interactive storytelling experience where viewers can participate in the narrative, essentially becoming part of the story rather than mere observers. This immersion fundamentally changes how stories are told and experienced.

5. **Democratization of Technology**: The speaker notes advancements in making volumetric filmmaking more accessible, such as developing techniques that allow for high-quality capture using less expensive equipment, like single cameras with AI assistance. This democratization is compared to the transition from film cameras to cell phone cameras, opening new possibilities for creators without significant budgets.

These points reflect the intersection of technology, storytelling, and future communication methods discussed in the video.

Export as PDF

X-TIMESTAMP-MAP:LOCAL=00:00:00.000,MPEGTS=900000

00:00.000 --> 00:01.285
It's actually interesting story.

00:01.465 --> 00:04.205
Um, I was always fascinated with, science fiction.

00:04.985 --> 00:06.285
And, actually,

00:06.395 --> 00:08.925
even though I was reading science fiction, I was not,

00:09.195 --> 00:11.565
incorporating it in my work when I was doing

00:11.565 --> 00:12.725
my masters in undergrad.

00:13.065 --> 00:15.205
And then I was in a conference as a PhD student,

00:15.345 --> 00:18.725
and I, met, person from Harvard,

00:19.105 --> 00:22.245
and she asked me in a conversation, she asked me, so,

00:22.245 --> 00:24.845
which science fiction book you are currently implementing?

00:25.565 --> 00:27.685
I was like, that's interesting way to look at it.

00:27.825 --> 00:30.245
So I said, well, I, I need to think about it.

00:30.545 --> 00:33.565
And I realized that actually I am implementing,

00:33.835 --> 00:35.045
star Trek holodeck.

00:35.065 --> 00:38.525
So this is what, what I'm doing is start a holodeck trying

00:38.545 --> 00:39.645
to create od.

00:39.915 --> 00:43.885
Then I kind of revised this motivation, to,

00:44.075 --> 00:48.165
more modern novel byte, Neil Stevenson called Diamond H,

00:48.865 --> 00:51.605
because in, in this novel, there is, the concept

00:51.665 --> 00:55.365
of reactive, which is like, interactive, film,

00:55.615 --> 00:57.525
which we, you can participate in.

00:57.825 --> 01:00.925
So I think that's, that's what motivated me from the,

01:01.075 --> 01:02.405
from the idea perspective.

01:02.865 --> 01:04.765
But there is also a vision behind it.

01:04.925 --> 01:08.165
I, I believe that, there is a word in which

01:08.705 --> 01:11.405
people can completely eliminate the distance.

01:11.945 --> 01:13.445
Um, we can,

01:13.445 --> 01:15.605
rather than communicating like we are right now,

01:15.635 --> 01:18.365
over zoom, very soon, I think we,

01:18.365 --> 01:19.965
within the next 10 years,

01:20.345 --> 01:23.645
we will be immersing ourselves in virtual wards

01:24.145 --> 01:27.085
and seeing each other as holograms, as we call it, as volume

01:27.645 --> 01:28.965
captured representations.

01:29.105 --> 01:30.445
And there'll be a common practice.

01:30.445 --> 01:31.485
It'll not be something novel.

01:31.615 --> 01:33.565
It'll be just the way we communicate instead

01:33.565 --> 01:36.045
of using a traditional 2D communication.

01:36.305 --> 01:37.765
And it's a great way to communicate

01:37.765 --> 01:40.685
because it, also,

01:41.285 --> 01:42.725
communicates the body language,

01:42.725 --> 01:45.165
through the 2D communication.

01:45.185 --> 01:46.805
You don't get that, that much.

01:46.925 --> 01:48.405
I mean, you can see maybe my gestures,

01:48.585 --> 01:52.525
but you can see that my body post, you don't, see like,

01:52.545 --> 01:53.645
if I'm relaxed

01:53.705 --> 01:57.165
or I'm stressed, you know, this, all this, little cues,

01:57.175 --> 01:59.125
which are important, are not being communicated.

01:59.425 --> 02:01.325
So I think this will be vastly improved.

02:01.425 --> 02:03.205
It will be almost like being in the same room.

02:03.705 --> 02:05.205
Uh, so that's another motivation.

02:05.385 --> 02:06.765
And in terms of, filmmaking,

02:06.925 --> 02:08.445
'cause as you know, I'm a both filmmaker

02:08.445 --> 02:09.445
and computer scientist.

02:09.665 --> 02:11.005
In terms of filmmaking, I,

02:11.165 --> 02:12.685
I think this is absolutely fascinating.

02:12.685 --> 02:14.765
What we can do. I, I will speak more about it,

02:14.825 --> 02:17.285
but it's just, just the idea that you can walk

02:17.285 --> 02:20.525
inside the film and become one of the actors for me, is,

02:20.585 --> 02:22.485
is enough to motivate me

02:22.485 --> 02:25.085
for 14 hours a day work on this technology.

02:25.185 --> 02:27.885
The volumetric capture technology is one

02:27.885 --> 02:30.205
of the technologies in something,

02:30.205 --> 02:32.285
which I call artificial reality continuum.

02:32.945 --> 02:35.125
The idea that you can augment reality

02:35.785 --> 02:37.445
or replace fragments of reality

02:37.475 --> 02:41.885
with a new artificially created, stimuli for any

02:41.885 --> 02:44.205
of your senses, not only for for visual sense,

02:44.205 --> 02:46.765
but for touch, for, smell, and so on.

02:47.185 --> 02:49.565
So, volume capture specifically allows you

02:49.565 --> 02:52.285
to record either objects and environments

02:52.385 --> 02:55.725
or actors, moving actors or people or,

02:56.065 --> 02:58.885
or, animals, in truly

02:59.015 --> 03:00.015
3D manner. So

03:00.015 --> 03:02.605
what I mean by that is it is not the fake 3D,

03:02.605 --> 03:04.965
which we have in 3D, films.

03:05.035 --> 03:06.765
When we go to cinema to watch a 3D film,

03:06.955 --> 03:09.885
what we are really getting is a 2D film, which was recorded

03:10.225 --> 03:11.485
for each eye separately.

03:11.825 --> 03:15.725
And we are, tricked to believe that this film is 3D

03:15.725 --> 03:19.085
volume capture is actually a, a recording of a 3D.

03:19.085 --> 03:20.565
So it records the volume

03:20.745 --> 03:22.605
or the surface of objects,

03:22.745 --> 03:25.045
so you can actually look at them from any perspective.

03:25.105 --> 03:27.565
So they become like life objects.

03:27.585 --> 03:28.805
The only difference is that

03:28.875 --> 03:30.845
with this atic capture technology, you cannot

03:31.065 --> 03:32.765
yet touch them, but you,

03:32.825 --> 03:34.205
that's why we call them holograms,

03:34.425 --> 03:36.885
but you can actually observe 'em from any perspective.

03:37.425 --> 03:40.085
So, the way it is done, you right now,

03:40.085 --> 03:41.605
you require a ring of cameras.

03:41.985 --> 03:43.965
So the actors will come into a special studio

03:43.965 --> 03:45.365
and there is a camera surrounding them.

03:45.705 --> 03:48.485
We have a studio like this, at American University.

03:48.745 --> 03:51.525
That's a rare, privilege, I would say

03:51.525 --> 03:53.125
because the studios are very expensive

03:53.145 --> 03:55.925
and only a few universities in the world, have them.

03:56.425 --> 03:58.965
Uh, we were lucky to get the National Science Foundation,

03:59.145 --> 04:00.285
funding for this.

04:00.625 --> 04:02.445
Uh, but interestingly enough, this,

04:02.445 --> 04:04.285
this technology is, democratizing.

04:04.505 --> 04:08.525
So, there is few companies including, myself,

04:08.865 --> 04:13.205
who are working on making, it possible to have,

04:13.555 --> 04:15.005
holograms of people recorded

04:15.005 --> 04:17.485
with a single camera using AI technologies.

04:17.505 --> 04:19.125
So instead of having a ring of cameras,

04:19.465 --> 04:21.805
you actually can record it with a cell phone camera,

04:21.905 --> 04:25.165
and yet get a full hologram, really truly 3D hologram.

04:25.165 --> 04:27.565
And whatever was not captured is recreated

04:27.565 --> 04:29.125
with generative AI techniques.

04:29.385 --> 04:31.165
So this is very close. It's coming.

04:31.455 --> 04:34.285
There are already prototypes, but it's not good quality yet.

04:34.435 --> 04:38.485
It's coming. And it'll open, new era of,

04:38.485 --> 04:39.525
volumetric filmmaking.

04:39.725 --> 04:41.845
'cause everybody will be able to do that.

04:41.915 --> 04:44.165
Like, when it change from 35 millimeter,

04:44.445 --> 04:47.205
millimeter camera, to a cell phone camera

04:47.385 --> 04:48.525
of a amazing quality,

04:48.825 --> 04:52.365
the same will happen when we can create easily without high

04:52.365 --> 04:54.005
costs, these volumetric experiences.

04:54.585 --> 04:57.205
So, volume capture is part of the spectrum of technologies,

04:57.465 --> 05:01.685
specifically referring to capturing moving actors,

05:01.945 --> 05:03.685
in a way that it's truly 3D

05:03.755 --> 05:04.965
this is an interesting topic.

05:05.665 --> 05:08.005
Uh, the reason why you would even go

05:08.005 --> 05:10.365
through all this trouble of using these new technologies

05:10.465 --> 05:14.525
and have, actors recorded in three, 3D

05:14.705 --> 05:16.965
and reconstruct the space where they are in

05:16.985 --> 05:18.725
or constructed in virtual reality.

05:19.265 --> 05:21.525
The reason why I would want to do it is

05:21.525 --> 05:23.645
because volumetric filmmaking,

05:23.645 --> 05:26.605
it's an entirely different way of experiencing story.

05:26.945 --> 05:28.125
It has nothing to do.

05:28.125 --> 05:30.205
There is no almost no similarities

05:30.205 --> 05:31.405
to traditional filmmaking.

05:31.755 --> 05:33.205
What is the main difference is

05:33.205 --> 05:35.005
that you are immersing yourself in the story.

05:35.005 --> 05:37.405
You're not doing a suspension of disbelief

05:37.635 --> 05:39.885
through the way the director told you the story.

05:40.105 --> 05:42.685
It is more like entering the film itself, like going

05:42.685 --> 05:44.085
to a theater, to a live theater.

05:44.705 --> 05:47.805
And, so basically what what we mean

05:47.805 --> 05:50.805
by interaction modes in volumetric storytelling,

05:51.345 --> 05:52.565
is everything.

05:52.875 --> 05:55.125
Some people say, interaction means like touching

05:55.125 --> 05:56.645
objects, interacting with objects.

05:57.195 --> 05:59.805
Interaction means, in this case, being able

05:59.865 --> 06:01.765
to control the view of the world.

06:01.865 --> 06:03.565
So you are walking around in this view,

06:03.745 --> 06:06.365
and you decide, not the director, not the cinematographer,

06:06.745 --> 06:08.045
but you decide whether you want

06:08.045 --> 06:10.965
to observe something from a closeup by co coming close to

06:10.965 --> 06:12.605
that or, from a distance.

06:13.065 --> 06:14.085
So that's one thing.

06:14.185 --> 06:17.605
The, the, the passive observation becomes interaction.

06:17.865 --> 06:19.005
And then you can go from there.

06:19.105 --> 06:20.445
So you can have an embodiment,

06:20.445 --> 06:22.285
so you can see, actually see your body.

06:22.665 --> 06:24.405
And that is also a form of interaction,

06:24.405 --> 06:26.765
because now your hands are the virtual hands of, let's say,

06:26.765 --> 06:27.805
an actor in that story.

06:28.145 --> 06:31.045
And you can see yourself as in, for example, in case

06:31.045 --> 06:33.725
of my work as Vera, one of the,

06:34.275 --> 06:35.525
main characters in the story,

06:35.935 --> 06:37.365
which I will be talking about later.

06:37.945 --> 06:41.165
Um, but, that, or that's also form of interaction.

06:41.175 --> 06:44.925
There is also, the interaction, with objects.

06:44.925 --> 06:47.725
So you can, it's not, necessary,

06:47.745 --> 06:49.885
but you can have objects which are interactive,

06:49.885 --> 06:51.445
so you can touch them, you can switch on the radio,

06:51.445 --> 06:53.645
switch off the radio, grab an apple.

06:53.825 --> 06:55.165
You can not yet eat the apple,

06:55.305 --> 06:58.445
but soon maybe we'll have a ability to eat the virtual apple

06:58.465 --> 07:00.685
and, get a taste of it.

07:00.685 --> 07:02.205
There are already experiments with it.

07:02.305 --> 07:05.285
So interaction means basically anything which is interaction

07:05.285 --> 07:06.685
mode is anything,

07:06.685 --> 07:08.885
which you can do in this experimental

07:08.955 --> 07:10.245
immersive environments.

07:10.555 --> 07:12.405
This actually entirely different medium.

07:12.595 --> 07:15.445
It's, it's as dissimilar as, let's say,

07:15.445 --> 07:16.605
theater and, and film.

07:17.145 --> 07:20.925
Um, what I mean is, is literally, you are losing,

07:21.265 --> 07:23.765
and then need to discover,

07:23.825 --> 07:25.805
100 plus years of cinematography.

07:25.825 --> 07:27.565
100 plus years of montage theory.

07:27.915 --> 07:31.965
Nothing from the traditional film translates into the,

07:32.505 --> 07:33.805
volumetric film.

07:33.805 --> 07:35.565
So, for example, there is no concept of cut.

07:35.755 --> 07:39.085
There's no concept of, any forms of montage.

07:39.085 --> 07:40.125
There's no concept of frame.

07:40.345 --> 07:43.205
You don't have, shot, you know, closeup or,

07:43.265 --> 07:44.885
or a medium, shot.

07:45.305 --> 07:47.525
All of this is controlled by the viewer.

07:48.065 --> 07:50.605
So the experience is entirely different.

07:50.945 --> 07:55.325
Now, 360 video is very, in my opinion, more similar to,

07:55.325 --> 07:57.805
traditional film because you, the director,

07:57.805 --> 08:00.245
is still in control of the position of a camera, and

08:00.245 --> 08:02.445
therefore the viewer at all times,

08:02.445 --> 08:04.205
the viewer cannot walk inside the film.

08:04.305 --> 08:06.285
It can be moved inside the film with the camera,

08:06.425 --> 08:09.005
and the camera is moving, but it cannot explore the space.

08:09.075 --> 08:10.525
It's not truly 3D.

08:10.825 --> 08:13.045
Uh, it already has some, some differences.

08:13.105 --> 08:15.485
For example, the framing, it doesn't, doesn't exist anymore.

08:15.485 --> 08:17.045
It's not a rectangle, it's a sphere.

08:17.585 --> 08:21.365
But it's, it's vastly there is a much greater difference

08:21.365 --> 08:22.405
between 360 video

08:22.985 --> 08:25.645
and, volumetric, film than there is

08:25.645 --> 08:26.725
between traditional film

08:26.725 --> 08:28.285
and 360 video, 360 videos,

08:28.315 --> 08:30.765
like having a very large screen around you.

08:31.025 --> 08:33.725
Uh, immersive, volumetric film is,

08:33.835 --> 08:35.085
something completely different.

08:35.485 --> 08:39.125
I, I think about it more as a, a keen a, a menu mackin

08:39.125 --> 08:42.205
to theater or, even better participatory theater

08:42.415 --> 08:46.325
where you are present in a, arranged scene.

08:46.465 --> 08:49.205
And you can either be an observer, passive observer,

08:49.505 --> 08:50.805
or you can be embodied

08:51.225 --> 08:54.445
and participate as an active, participant in the story.

08:54.785 --> 08:58.405
And I want to stress here one important, issue

08:58.405 --> 09:01.645
because many people identify immersive filmmaking or,

09:01.665 --> 09:03.125
or, volumetric filmmaking

09:03.155 --> 09:05.605
with something similar to computer games.

09:06.025 --> 09:09.085
But the difference is that there is still linearity in

09:09.085 --> 09:10.125
the immersive filmmaking.

09:10.125 --> 09:14.205
The story is still told in a traditional, sense, meaning

09:14.205 --> 09:16.565
that there can be an arc, of the story.

09:16.945 --> 09:20.125
And there is the, the participation does not necessarily,

09:20.265 --> 09:22.805
it can, but usually it doesn't influence the story.

09:22.825 --> 09:25.205
So it's not a nonlinear, storytelling.

09:25.435 --> 09:27.605
It's, it's different than interactive, films.

09:27.705 --> 09:30.125
We now see where we can choose your, own adventure.

09:30.715 --> 09:34.525
It's still, a narrative as a theater play.

09:35.025 --> 09:36.685
Um, but so it's not changed.

09:36.685 --> 09:37.725
The, the story is not changed,

09:37.865 --> 09:39.965
but you are participating in it actively.

09:39.965 --> 09:41.165
And that's, that's the key here.

09:41.625 --> 09:43.485
Uh, it is not a suspension of this belief.

09:43.485 --> 09:46.565
It's an immersion. Vera was one of the first,

09:46.815 --> 09:48.365
volumetric films ever recorded,

09:48.465 --> 09:52.765
and it was also the first, interactive volumetric field.

09:53.185 --> 09:56.365
Um, what we did there, basically what I described as a,

09:56.565 --> 09:58.685
a result of, of this experiment, which is the,

09:58.685 --> 10:01.125
the finding out that the medium is entirely different

10:01.155 --> 10:02.285
than filmmaking.

10:02.305 --> 10:04.205
My background is in filmmaking and computer science.

10:04.265 --> 10:06.925
So I approached this initially when I,

10:07.035 --> 10:09.365
learned about the technology and started to build them.

10:09.925 --> 10:14.485
I approached it, as the old, media, media,

10:14.485 --> 10:17.405
creator would usually approach the new medium by, by trying

10:17.465 --> 10:20.445
to kind of translate what is possible in this new medium.

10:20.865 --> 10:22.485
Um, and then I quickly discovered that,

10:22.485 --> 10:23.525
that it's a wrong approach.

10:23.545 --> 10:26.325
We need an experiment, which will show us what,

10:26.395 --> 10:27.805
what is different, right?

10:27.865 --> 10:29.365
So what we came up with is

10:29.365 --> 10:31.565
to take a Katherine Mansfield short,

10:31.575 --> 10:32.645
story called D Pickle.

10:32.755 --> 10:35.485
It's a 1920 story, by a,

10:35.685 --> 10:37.605
a writer Katherine Mansfield from New Zealand.

10:38.025 --> 10:42.685
And we, recorded the same story in various, media.

10:42.825 --> 10:44.245
So the first one was the story

10:44.245 --> 10:45.405
stuff, so you could just read it.

10:45.785 --> 10:47.565
The second one was the radio play,

10:47.565 --> 10:49.245
so no visuals, that's radio play.

10:49.585 --> 10:51.365
The third one was traditional film.

10:51.365 --> 10:54.725
So it's a 15, 20 minute, depending on a version film,

10:54.735 --> 10:56.725
which happens in a, one location.

10:57.225 --> 11:00.605
And it basically uses all, even over users specifically

11:00.605 --> 11:03.965
for this purpose, all the traditional techniques of,

11:04.065 --> 11:05.525
cinematography and montage.

11:06.065 --> 11:08.125
And then we recorded the 360 video

11:08.185 --> 11:09.365
to see how different it is.

11:09.945 --> 11:13.365
And then finally, what we did was an interactive version of

11:14.145 --> 11:17.645
the same story when actors were recorded holographically.

11:18.065 --> 11:20.365
And you can enter that story, walk

11:20.365 --> 11:22.965
around the set in virtual reality,

11:23.545 --> 11:25.445
or actually in augmented reality too,

11:25.825 --> 11:28.005
and observe the actors interacting

11:28.005 --> 11:29.565
with each other the way they want,

11:29.865 --> 11:31.325
and what is the most interesting and,

11:31.325 --> 11:32.365
and the most novel there.

11:32.585 --> 11:33.925
You can become one of the actors.

11:34.065 --> 11:35.365
So you can become Vera

11:35.625 --> 11:37.925
and literally tell the lines of, say the lines,

11:38.125 --> 11:42.285
Vera says in the story to the, hologram

11:42.545 --> 11:45.325
of an actor, who plays against you,

11:45.385 --> 11:46.645
so plays with you, right?

11:46.745 --> 11:50.005
So this be this, this basically makes the story,

11:50.195 --> 11:53.605
very personal because you are embodied in the Veras body.

11:53.745 --> 11:55.805
You can, you say what Vera says,

11:55.865 --> 11:58.285
you can see the lines if you don't remember them,

11:58.285 --> 12:01.005
or you can learn them and act the whole, play

12:01.345 --> 12:04.005
or the whole, film, by yourself.

12:04.225 --> 12:06.165
And then there is a reaction from the hologram,

12:06.165 --> 12:08.325
which responds to that, and it's still linear.

12:08.325 --> 12:10.485
So it's not like, depending what you say,

12:10.485 --> 12:11.485
something else will happen.

12:11.785 --> 12:14.925
You are supposed to say specific lines from the story,

12:15.305 --> 12:16.645
and then you'll get a response,

12:16.905 --> 12:18.685
but it still feels very immersive.

12:18.865 --> 12:20.725
Uh, it, it also comes from the fact that I used

12:20.725 --> 12:23.165
to be a theater actor as a, a teenager.

12:23.345 --> 12:25.365
So it basically, I love this experience,

12:25.545 --> 12:27.925
but I didn't have access recently to theater,

12:27.945 --> 12:30.285
and I'm an immigrant, so my accent is very strong,

12:30.285 --> 12:33.325
very difficult to, to get to theater in us, for me.

12:33.865 --> 12:37.205
So, I basically created the system when you can,

12:37.555 --> 12:40.645
play in a movie, as if you were an actor

12:40.705 --> 12:43.405
or play in a theater, as if you were an actor in the,

12:43.615 --> 12:45.685
let's say, comfort of your own VR headset.

12:46.665 --> 12:50.525
Um, yeah. So, so that's, that's the, story of Vera.

12:50.675 --> 12:51.965
It's a transmedia in a sense

12:51.965 --> 12:54.765
that you can explore this story from different perspectives.

12:54.815 --> 12:56.565
There are many other great films.

12:56.625 --> 12:59.085
And recently what happens is that,

12:59.365 --> 13:01.565
I think the best place to explore it is Tribe Beka Film

13:01.725 --> 13:04.805
Festival, and, Venice Film Festival, Venice,

13:04.805 --> 13:08.165
even dedicated entire island to, volumetric

13:08.165 --> 13:09.365
and VR experiences.

13:09.745 --> 13:13.205
And in recent years, the quality is going up tremendously.

13:13.205 --> 13:16.205
It's amazing what people are coming out, up with.

13:16.275 --> 13:19.165
It's, it's like 1920s or thirties in film.

13:19.235 --> 13:22.245
It's, it's getting that so many things are discovered,

13:22.705 --> 13:24.445
I'm thinking about writing a book about it

13:24.445 --> 13:25.965
because it's, it's so fascinating.

13:25.965 --> 13:29.125
There's enough material to, to cover all these,

13:29.155 --> 13:32.325
innovations in, in storytelling in volumetric film.

13:32.785 --> 13:36.645
We are literally seeing a birth of a new medium,

13:36.645 --> 13:38.245
which will be entirely,

13:38.245 --> 13:39.805
different way of telling the stories.

13:40.145 --> 13:43.205
And it doesn't mean that I think the cinema will die

13:43.305 --> 13:44.845
or, will be even less popular.

13:45.275 --> 13:46.805
It's, it's just entirely different.

13:46.835 --> 13:48.965
It's like us cinema didn't kill theater.

13:49.545 --> 13:51.965
Um, volumetric filmmaking will not,

13:52.265 --> 13:55.725
reduce the interest in, in, traditional films.

13:56.055 --> 14:00.605
It'll just create one, yet another,

14:01.165 --> 14:04.325
technique, for expressing yourself and expressing and,

14:04.505 --> 14:06.165
and, and creating stories.

14:06.545 --> 14:10.765
And it's, it's a, it's the most so far immersive except

14:10.785 --> 14:13.485
for a personal, presence in a theater,

14:13.795 --> 14:16.165
something which we many people don't experience anymore,

14:16.165 --> 14:18.805
because so few people go to theater, compared to,

14:19.455 --> 14:21.245
let's say, Italian opera.

14:21.345 --> 14:24.085
And everybody in masses were going to the Italian opera in,

14:24.425 --> 14:26.405
a hundred years ago.

14:26.705 --> 14:29.725
Um, I think it will be back in a sense that we'll be able

14:29.745 --> 14:32.965
to, participate in this very,

14:32.965 --> 14:34.565
immersive, stories.

14:35.185 --> 14:38.925
But, using the, fear headset.

14:39.505 --> 14:42.005
Now, the important thing is that, that the technology has

14:42.005 --> 14:43.005
to become more accessible.

14:43.025 --> 14:45.165
It cannot cost the $1 million to have a studio.

14:45.625 --> 14:48.725
And it also cannot be that as, as everyone was said, it,

14:48.825 --> 14:50.765
you, you are, you are not supposed to put a TV

14:50.785 --> 14:52.605
On your head and then look at it.

14:52.905 --> 14:54.845
Uh, right now the VR headsets are,

14:54.845 --> 14:57.605
basically two TV screens put on your head.

14:57.745 --> 15:02.285
Uh, it's changing. I, I, I have a, apple Vision Pro,

15:02.345 --> 15:04.525
and it's comparably better than the previous,

15:04.525 --> 15:05.765
headsets, but we are still not there.

15:06.265 --> 15:09.125
So what I envision is an ambient vr.

15:09.265 --> 15:13.285
So, a vr which will not require any, equipment,

15:13.385 --> 15:15.805
to be worn, basically would be projected on retina.

15:16.385 --> 15:18.205
Either it can be either done.

15:18.345 --> 15:21.045
Um, and at least for now with glasses like this,

15:21.045 --> 15:23.045
so lightweight glasses that still needs to be worn.

15:23.145 --> 15:26.005
And then it'll be, contact lenses,

15:26.005 --> 15:28.405
which already exist, but in very resolution

15:28.705 --> 15:31.605
or less projection from a distance on your director on your

15:31.605 --> 15:33.285
retina, which doesn't require anything.

15:33.545 --> 15:35.085
So once these kind of technologies

15:35.185 --> 15:36.965
and this, these are quite far away,

15:36.965 --> 15:38.245
they're not on the market

15:38.265 --> 15:41.005
or close to market yet, when these technologies exist,

15:41.365 --> 15:43.405
I think this will be a golden age for,

15:43.405 --> 15:44.885
immersive storytelling

15:44.885 --> 15:47.285
and me immersive anything, immersive meetings, immersive,

15:47.465 --> 15:49.165
schools, immersive worker

15:49.285 --> 15:51.445
environments, training, and so on.

15:51.585 --> 15:53.125
But for storytelling, it will be a

15:53.125 --> 15:54.885
wonderful place to explore.

15:54.945 --> 15:57.485
So, I'm very excited about the future, especially

15:57.485 --> 16:02.245
with the hope, hopes and, and risks of ai,

16:02.265 --> 16:06.045
and generative AI helping, artists to achieve

16:06.045 --> 16:08.045
that without having millions of dollars of budget.

Quick Answers

Exploring the Future of Immersive Storytelling

00:00:00 - It's actually interesting story. I was always fascinated with, science fiction. And, actually, even though I was reading science fiction, I was not, incorporating it in my work when I was doing my masters in undergrad. And then I was in a conference as a PhD student, and I, met, person from Harvard, and she asked me in a conversation, she asked me, so, which science fiction book you are currently implementing? I was like, that's interesting way to look at it. So I said, well, I, I need to think about it. And I realized that actually I am implementing,
00:00:35 - So this is what, what I'm doing is start a holodeck trying to create od. Then I kind of revised this motivation, to, more modern novel byte, Neil Stevenson called Diamond H, because in, in this novel, there is, the concept of reactive, which is like, interactive, film, which we, you can participate in. So I think that's, that's what motivated me from the, from the idea perspective. But there is also a vision behind it.
00:01:08 - people can completely eliminate the distance. we can, rather than communicating like we are right now, over zoom, very soon, I think we, within the next 10 years, we will be immersing ourselves in virtual wards and seeing each other as holograms, as we call it, as volume captured representations. And there'll be a common practice. It'll not be something novel. It'll be just the way we communicate instead of using a traditional 2D communication.
00:01:37 - because it, also, communicates the body language, through the 2D communication. You don't get that, that much. I mean, you can see maybe my gestures, but you can see that my body post, you don't, see like, if I'm relaxed or I'm stressed, you know, this, all this, little cues, which are important, are not being communicated. So I think this will be vastly improved. It will be almost like being in the same room. so that's another motivation. And in terms of, filmmaking,
00:02:08 - and computer scientist. In terms of filmmaking, I, I think this is absolutely fascinating. What we can do. I, I will speak more about it, but it's just, just the idea that you can walk inside the film and become one of the actors for me, is, is enough to motivate me for 14 hours a day work on this technology. The volumetric capture technology is one of the technologies in something, which I call artificial reality continuum. The idea that you can augment reality or replace fragments of reality
00:02:41 - of your senses, not only for for visual sense, but for touch, for, smell, and so on. So, volume capture specifically allows you to record either objects and environments or actors, moving actors or people or, or, animals, in truly 3D manner. So what I mean by that is it is not the fake 3D, which we have in 3D, films. When we go to cinema to watch a 3D film, what we are really getting is a 2D film, which was recorded

Revolutionizing Filmmaking with Volumetric Capture

00:03:11 - And we are, tricked to believe that this film is 3D volume capture is actually a, a recording of a 3D. So it records the volume or the surface of objects, so you can actually look at them from any perspective. So they become like life objects. The only difference is that with this atic capture technology, you cannot yet touch them, but you, that's why we call them holograms, but you can actually observe 'em from any perspective. So, the way it is done, you right now, you require a ring of cameras.
00:03:43 - and there is a camera surrounding them. We have a studio like this, at American University. That's a rare, privilege, I would say because the studios are very expensive and only a few universities in the world, have them. we were lucky to get the National Science Foundation, funding for this. but interestingly enough, this, this technology is, democratizing. So, there is few companies including, myself, who are working on making, it possible to have,
00:04:15 - with a single camera using AI technologies. So instead of having a ring of cameras, you actually can record it with a cell phone camera, and yet get a full hologram, really truly 3D hologram. And whatever was not captured is recreated with generative AI techniques. So this is very close. It's coming. There are already prototypes, but it's not good quality yet. It's coming. And it'll open, new era of, volumetric filmmaking. 'cause everybody will be able to do that. Like, when it change from 35 millimeter,
00:04:47 - of a amazing quality, the same will happen when we can create easily without high costs, these volumetric experiences. So, volume capture is part of the spectrum of technologies, specifically referring to capturing moving actors, in a way that it's truly 3D this is an interesting topic. the reason why you would even go through all this trouble of using these new technologies and have, actors recorded in three, 3D
00:05:16 - or constructed in virtual reality. The reason why I would want to do it is because volumetric filmmaking, it's an entirely different way of experiencing story. It has nothing to do. There is no almost no similarities to traditional filmmaking. What is the main difference is that you are immersing yourself in the story. You're not doing a suspension of disbelief through the way the director told you the story. It is more like entering the film itself, like going to a theater, to a live theater.
00:05:47 - by interaction modes in volumetric storytelling, is everything. Some people say, interaction means like touching objects, interacting with objects. Interaction means, in this case, being able to control the view of the world. So you are walking around in this view, and you decide, not the director, not the cinematographer, but you decide whether you want to observe something from a closeup by co coming close to that or, from a distance. So that's one thing. The, the, the passive observation becomes interaction.

Exploring Embodiment in Volumetric Storytelling

00:06:19 - So you can have an embodiment, so you can see, actually see your body. And that is also a form of interaction, because now your hands are the virtual hands of, let's say, an actor in that story. And you can see yourself as in, for example, in case of my work as Vera, one of the, main characters in the story, which I will be talking about later. but, that, or that's also form of interaction. There is also, the interaction, with objects. So you can, it's not, necessary, but you can have objects which are interactive,
00:06:51 - switch off the radio, grab an apple. You can not yet eat the apple, but soon maybe we'll have a ability to eat the virtual apple and, get a taste of it. There are already experiments with it. So interaction means basically anything which is interaction mode is anything, which you can do in this experimental immersive environments. This actually entirely different medium. It's, it's as dissimilar as, let's say, theater and, and film. what I mean is, is literally, you are losing,
00:07:23 - 100 plus years of cinematography. 100 plus years of montage theory. Nothing from the traditional film translates into the, volumetric film. So, for example, there is no concept of cut. There's no concept of, any forms of montage. There's no concept of frame. You don't have, shot, you know, closeup or, or a medium, shot. All of this is controlled by the viewer. So the experience is entirely different. Now, 360 video is very, in my opinion, more similar to,
00:07:57 - is still in control of the position of a camera, and therefore the viewer at all times, the viewer cannot walk inside the film. It can be moved inside the film with the camera, and the camera is moving, but it cannot explore the space. It's not truly 3D. it already has some, some differences. For example, the framing, it doesn't, doesn't exist anymore. It's not a rectangle, it's a sphere. But it's, it's vastly there is a much greater difference between 360 video and, volumetric, film than there is
00:08:26 - and 360 video, 360 videos, like having a very large screen around you. immersive, volumetric film is, something completely different. I, I think about it more as a, a keen a, a menu mackin to theater or, even better participatory theater where you are present in a, arranged scene. And you can either be an observer, passive observer, or you can be embodied and participate as an active, participant in the story. And I want to stress here one important, issue
00:09:01 - or, volumetric filmmaking with something similar to computer games. But the difference is that there is still linearity in the immersive filmmaking. The story is still told in a traditional, sense, meaning that there can be an arc, of the story. And there is the, the participation does not necessarily, it can, but usually it doesn't influence the story. So it's not a nonlinear, storytelling. It's, it's different than interactive, films. We now see where we can choose your, own adventure.

Immersive Interaction in Volumetric Storytelling

00:09:35 - but so it's not changed. The, the story is not changed, but you are participating in it actively. And that's, that's the key here. it is not a suspension of this belief. It's an immersion. Vera was one of the first, volumetric films ever recorded, and it was also the first, interactive volumetric field. what we did there, basically what I described as a, a result of, of this experiment, which is the, the finding out that the medium is entirely different
00:10:02 - My background is in filmmaking and computer science. So I approached this initially when I, learned about the technology and started to build them. I approached it, as the old, media, media, creator would usually approach the new medium by, by trying to kind of translate what is possible in this new medium. and then I quickly discovered that, that it's a wrong approach. We need an experiment, which will show us what, what is different, right? So what we came up with is to take a Katherine Mansfield short,
00:10:32 - It's a 1920 story, by a, a writer Katherine Mansfield from New Zealand. And we, recorded the same story in various, media. So the first one was the story stuff, so you could just read it. The second one was the radio play, so no visuals, that's radio play. The third one was traditional film. So it's a 15, 20 minute, depending on a version film, which happens in a, one location. And it basically uses all, even over users specifically for this purpose, all the traditional techniques of,
00:11:06 - And then we recorded the 360 video to see how different it is. And then finally, what we did was an interactive version of the same story when actors were recorded holographically. And you can enter that story, walk around the set in virtual reality, or actually in augmented reality too, and observe the actors interacting with each other the way they want, and what is the most interesting and, and the most novel there. You can become one of the actors.
00:11:35 - and literally tell the lines of, say the lines, Vera says in the story to the, hologram of an actor, who plays against you, so plays with you, right? So this be this, this basically makes the story, very personal because you are embodied in the Veras body. You can, you say what Vera says, you can see the lines if you don't remember them, or you can learn them and act the whole, play or the whole, film, by yourself.
00:12:06 - which responds to that, and it's still linear. So it's not like, depending what you say, something else will happen. You are supposed to say specific lines from the story, and then you'll get a response, but it still feels very immersive. it, it also comes from the fact that I used to be a theater actor as a, a teenager. So it basically, I love this experience, but I didn't have access recently to theater, and I'm an immigrant, so my accent is very strong, very difficult to, to get to theater in us, for me. So, I basically created the system when you can,

Emerging Innovations in Volumetric Filmmaking

00:12:40 - or play in a theater, as if you were an actor in the, let's say, comfort of your own VR headset. yeah. So, so that's, that's the, story of Vera. It's a transmedia in a sense that you can explore this story from different perspectives. There are many other great films. And recently what happens is that, I think the best place to explore it is Tribe Beka Film Festival, and, Venice Film Festival, Venice, even dedicated entire island to, volumetric
00:13:09 - And in recent years, the quality is going up tremendously. It's amazing what people are coming out, up with. It's, it's like 1920s or thirties in film. It's, it's getting that so many things are discovered, I'm thinking about writing a book about it because it's, it's so fascinating. There's enough material to, to cover all these, innovations in, in storytelling in volumetric film. We are literally seeing a birth of a new medium, which will be entirely,
00:13:40 - And it doesn't mean that I think the cinema will die or, will be even less popular. It's, it's just entirely different. It's like us cinema didn't kill theater. volumetric filmmaking will not, reduce the interest in, in, traditional films. It'll just create one, yet another, technique, for expressing yourself and expressing and, and, and creating stories. And it's, it's a, it's the most so far immersive except
00:14:13 - something which we many people don't experience anymore, because so few people go to theater, compared to, let's say, Italian opera. And everybody in masses were going to the Italian opera in, a hundred years ago. I think it will be back in a sense that we'll be able to, participate in this very, immersive, stories. But, using the, fear headset. Now, the important thing is that, that the technology has
00:14:43 - It cannot cost the $1 million to have a studio. And it also cannot be that as, as everyone was said, it, you, you are, you are not supposed to put a TV On your head and then look at it. right now the VR headsets are, basically two TV screens put on your head. it's changing. I, I, I have a, apple Vision Pro, and it's comparably better than the previous, headsets, but we are still not there. So what I envision is an ambient vr. So, a vr which will not require any, equipment,
00:15:16 - Either it can be either done. and at least for now with glasses like this, so lightweight glasses that still needs to be worn. And then it'll be, contact lenses, which already exist, but in very resolution or less projection from a distance on your director on your retina, which doesn't require anything. So once these kind of technologies and this, these are quite far away, they're not on the market or close to market yet, when these technologies exist, I think this will be a golden age for,

Exciting Future of Immersive Storytelling

00:16:06 - and me immersive anything, immersive meetings, immersive, schools, immersive worker environments, training, and so on. But for storytelling, it will be a wonderful place to explore. So, I'm very excited about the future, especially with the hope, hopes and, and risks of ai, and generative AI helping, artists to achieve that without having millions of dollars of budget.

Faculti AI can make mistakes

Research

Policy

Analysis

Data

Originals

Explainers

Podcasts

Watch the latest episode of The Exchange – the show where influential policymakers and leading researchers engage in insightful discussions shaping the future.

Episode: Brazil's Economic Resilience