Virtual model box

This is the archive post page

About virtual model box

The Virtual Model Box (VMB) is a new tool designed to improve and expand the workflow
of stage design, lighting design, and introduce creatives to the company’s venues. This
tool can visualise sets in vastly higher fidelity than previously achieved in-house, and
accurately render complex visuals such as reflections, haze, practical effects and lighting.
VMB has a suite of easy to use tools that should aid in some of the common problems
faced today by new artists and designers that are unfamiliar with the RSC’s theatre spaces.
These tools also provide a method to create and modify white card style environments via
the placement and modification of virtual props, performers, lights, and cameras.
The VMB features an accurate layout of audience seating in the venues which can be
easily updated by the development team to new layouts as they come. These seats allow
for easy testing of different eyelines, checking accessibility screens are easily viewed, and
can also be used to generate promotional and advertising opportunities such as seat view
images for websites.

The VMB has been designed to fit into the existing pipeline of the Drafting team and can
ingest digital white cards rapidly and absorb them into the VMB toolkit, making them
available to use in the VMB tool.

Beyond the current suite of tools, VMB is designed to be expanded upon, with a strong
focus on expansion to feedback from those who will use the tool, and be fluid and flexible
for different use cases. The tool is built on the Unreal Engine, which is a free software and
gives near unlimited options for expansion.

What have we achieved so far?

A digital model of the Royal Shakespeare Theatre from the gallery.

Virtual Seating Visualisation

Our team has successfully managed to import precise seating blueprints sourced from the drafting teams, subsequently converting these into digital replicas of venue seating configurations, and aligning them accurately within the virtual environment. This advancement facilitates the visualisation of the stage from any given seat within the venue, thus presenting a superior planning utility for orchestrating shows.
Additionally, it offers the capability to generate realistic imagery depicting the view an audience member will have from a specific seat, a feature that holds promise for augmenting box-office sales through enriched booking experiences. Through this technology, a seamless bridge between physical and digital
venue representation has been established, offering a robust tool for both planners and patrons to
interact with the venue in a virtual capacity prior to physical attendance. This not only enhances the
planning and booking experience but also sets the stage for a more immersive engagement with
the venue’s offerings.

Pipeline Integration with Drafting Teams

We have devised a bespoke pipeline in collaboration with the drafting department, preserving their existing workflow whilst assimilating set layouts into our digital stage. Our system is adept at importing various file types such as STEP or GLTF, facilitating a fully automated import and setup process of the digital set through a few straightforward steps. This development has significantly reduced the complexity threshold for any team member to import assets and preview them on stage. Through this streamlined process, we’ve managed to foster a more inclusive environment where individuals, regardless of technical expertise, can contribute to the staging process, thereby expediting the project workflow and enhancing the overall efficiency.

Interactive Toolkit for Stage-Previsualisation
We have engineered a customised user interface wherein a designer can trial various elements including props, set pieces, effects, and more on the digital stage, with an extensive control over their placement, rotation, and scale. This toolkit also accommodates pre-animated elements, enabling the examination of automation or other animated content within the virtual stage environment. This system is seamlessly integrated with our seating tools, permitting the designer to evaluate these elements from any vantage point within the venue. Through this fusion, designers are afforded a comprehensive platform to meticulously plan and preview stage setups, ensuring an optimised audience experience from every seat in the house.

R&D Process
To cultivate these tools, we have mobilised team members across various departments, tailoring the tools to meet the distinct requisites inherent within each department. We have adopted an iterative methodology towards this endeavour, envisaging the toolkit not as a static entity, but as an evolving asset poised to mature and diversify over time in alignment with the unfolding needs of the team. Utilising source control, we have established a mechanism to share the project, enabling the provision of frequent updates as necessitated. This collaborative and adaptive approach not only expedites the development cycle but also ensures a robust, up-to-date toolkit that resonates with the dynamic demands of the project and the team.

2020/21 LiDAR scan of the RSC Stratford Upon Avon campus takes place
2021/22 Porting of Lidar scan into Unreal Engine, rebuilding furniture (doors, seats)
2022/23 Interactive functionality developed, Seat view preview – Swan Launch
2023/24 VMB hosted within drawing office for beta testing (Tempest 2023) + ongoing

Findings

We have identified numerous additional features intended for integration into the
Virtual Model Box (VMB); a selection of key elements include:

Virtual Lighting Technical-Visualisation:
We aim to harness the robust lighting and shadow system of Unreal Engine, namely Lumen and Virtual Shadow Maps, as a medium to examine lighting setups, thereby enhancing the planning phase of show lighting. This system is envisioned to dovetail seamlessly with existing lighting pre-visualisation tools, enabling the import of those setups to preserve the established workflow of respective departments, whilst introducing new avenues to test those setups within our fully adorned digital stage. By leveraging DMX protocols, we aspire to significantly expedite the show’s lighting planning process, initiating this crucial phase much earlier than currently feasible, well in advance of the real venue’s availability. This approach not only augments the precision and efficacy of lighting design but also broadens the temporal window for meticulous planning and adjustments, ensuring an optimised lighting experience come the actual event.

Flexible Shaders for Material Testing:
We aim to broaden the scope of user interaction by enabling the testing of diverse materials on the digital set, thereby furnishing a visual platform for exploring alternative material choices for set construction. This initiative, which transcends the capabilities of existing solutions, is poised to significantly enrich the creative approach to set design. To realise this ambition, we intend to amass a substantial library of material options, coupled with a user-friendly interface furnished with intuitive filters and search functionality. This interactive and expansive material testing environment not only facilitates a more informed decision-making process but also nurtures a conducive space for creative experimentation, driving the design process towards more innovative and visually appealing set constructions.

Long Term Plans
The long-term trajectory of this project unveils numerous avenues ripe for exploration. Initially conceived as a previsualisation tool, it aids in the meticulous design and development of stagecraft. However, we envisage extending its utility to the audience. With a digitally constructed stage and its comprehensive asset repository, we are on the cusp of actualising a digital replica of the final show. This venture paves the way for extending show access to digital audiences, thereby amplifying reach and embellishing the show with added features like enhanced accessibility and multilingual options. Furthermore, we recognise the burgeoning affinity for digital and interactive experiences, particularly among younger generations. The Virtual Model Browser (VMB) holds the potential to morph into an engaging platform for interactive learning or gaming experiences. This evolution not only significantly amplifies accessibility but also unfolds multilingual opportunities, addressing challenges often encountered in physical venues.
Through these envisioned expansions, VMB is poised to become a versatile tool that bridges the physical and digital realms, fostering enriched experiences for both show creators and audiences alike.

What are MetaHumans?

MetaHumans are essentially high-fidelity digital characters created using the MetaHuman Creator, a cloud-streamed app. They bridge the gap between cartoonish characters and fully photorealistic humans in digital environments. These characters are not just visually impressive but are also equipped with a range of realistic facial expressions and movements, bringing the technology closer to replicating live performance.

R&D processes

Over several years, we ran short sprints to explore the tools, technologies and formats for hybrid production. Working with our partners, creatives and technologists, this period of activity tested viable workflows which offer the potential to incorporate into RSC future productions.

Research Questions:

How can we manage real time live data feeds from actors on stage and serve across multiple channels with low latency & suitable QoS for creative work?
How can physical worlds (set, environments, objects, worlds) be successfully replicated digitally without affecting the live experience? How can those objects respond to interaction both live & digitally?
Where are the challenges and limitations of the current technology offering to achieve our aims?

We toured a demonstration of outcomes to SXSW festival (2022), Beyond Conference (2022) and, more recently at Mozfest, Mozilla Foundation’s convening in Amsterdam (2024) which you can watch online below:

How has The RSC used performance capture and rigs in performance?

The RSC has used performance capture technology in previous productions and digital projects – from The Tempest in 2016 (Realtime avatar of Ariel on stage) and Dream 2021 (using mocap and facial data to enable performers to animate digital characters in game engine).

Capacities of MetaHumans

Visual Realism

With the MetaHuman Creator, users can design characters that range from absolutely lifelike to any imaginative variation. The skin, eyes, teeth, hair, and every detail are intricately designed. The tool allows for extensive customisation. From facial features, skin tones, hairstyles, and even imperfections.

Rigging and Animation

MetaHumans come with a predefined skeleton and facial rig, making it easier to animate them. With the integration of motion capture technology, their movements can mimic those of a real actor without considerable developer costs as previously required. As they are part of the Unreal Engine ecosystem, they can easily inport into games and other UE5 applications.

Motion Capture

Actors can wear motion capture suits and perform live. Their movements are then translated in real-time to their MetaHuman counterparts in the game environment. We can use this data as recorded performance, or enable performers to interact with audiences in game.

The blend of MetaHuman technology, motion capture, and real-time game engines like Unreal Engine have the potential to reshape the boundaries of what’s possible in live performance and gaming. As tools evolve and become more accessible, we can explore how and where we might use these tools in our creative work.

What is photogrammetry?

Photogrammetry involves extracting three-dimensional (3D) information from photographs. This technique encompasses capturing overlapping images of an object, person, structure, or space and converting them into 3D digital models. Aerial and close-range photogrammetry represent the two primary categories within this field.

Aerial photogrammetry entails utilising aircraft to capture aerial photographs that can subse- quently be transformed into 3D models or digitally mapped. Today, this process can also be accomplished using drones, which have significantly simplified the safe acquisition of data from challenging-to-reach or hazardous locations, where conventional surveying methods may be impractical or perilous.

Close-range photogrammetry, on the other hand, involves capturing images using a hand-held camera or a camera affixed to a tripod. The objective of this approach is not to generate topographic maps but rather to create 3D models of relatively smaller objects. The images are usually captured sequentially.

DSLR array scan, captures a object or person by taking multiple photo- graphs at once. An array of lights and cameras are triggered simultaneously, giving in a quick capture and consistent results.

What are the potential use cases for theatre companies?

Photogrammetry offers numerous commercial and archiving uses for museums, galleries and Arts Institutions such as the RSC. Here are some potential applications demonstrating how photogrammetry can revolutionise the way institutions engage with audiences, preserve cultural heritage, and facilitate research and education in the digital age.:

Virtual Exhibitions: Photogrammetry enables the creation of immersive virtual exhibitions. Visitors can explore digital replicas of artworks and artefacts, providing access to a broader audience worldwide. This technology enhances outreach, education, and engagement, espe- cially for individuals who cannot physically visit the museum or gallery.

Online Collections: Museums can utilise photogrammetry to digitize their entire collection and create online databases. This allows users to browse and search through the collection remote- ly, providing a comprehensive digital archive for researchers, students, and enthusiasts.

Conservation and Restoration: Photogrammetry can aid in the documentation, conserva- tion, and restoration of artworks. High-resolution 3D models can capture fine details, which can be studied to better understand the piece, plan restoration processes, and track changes over time.

Reproduction and Merchandising: Photogrammetry can be used to create accurate replicas of artworks and artefacts for reproduction and merchandising purposes. These replicas can be sold in museum shops or used for educational purposes, enabling broader dissemination of cultural heritage.

Archival Preservation: 3D models created through photogrammetry can serve as archival records of valuable objects. In the event of damage, loss, or destruction, these models can act as references for restoration or reconstruction efforts.

Interactive Installations: Photogrammetry can support the creation of interactive installations within museums and galleries. Visitors can manipulate and explore 3D models through touch screens or virtual reality (VR) experiences, enhancing the overall visitor experience and offer- ing new perspectives on artworks and artefacts.

Research and Education: Photogrammetry enables detailed analysis and research on cultural objects without physically handling them. Scholars and researchers can study objects remotely, examining their intricate details, materials, and structures. Additionally, educational institutions can utilise 3D models for teaching purposes, providing students with hands-on experiences even when the objects are not physically accessible.

Documentation and Cataloguing: Photogrammetry simplifies the process of documenting and cataloguing objects in a museum or gallery. Instead of traditional manual methods, the creation of 3D models can streamline the inventory management process and improve the accuracy and efficiency of record-keeping.

Selling 3D Models on Game Marketplaces: Photogrammetry-generated 3D models have great potential for commercialisation in the gaming industry. Game developers and designers are constantly seeking high-quality assets to enhance their virtual worlds and provide realistic experi- ences to players. By selling 3D models on game marketplaces like Epic’s Unreal Engine Market- place, artists and museums can tap into this demand and monetize their digital replicas.

Asset Creation for Games: Photogrammetry allows for the creation of highly detailed and accurate 3D models that can be used as assets in video games. These models can be textures, characters, objects, or environments, providing game developers with ready-to-use content that saves them time and resources in the asset creation pipeline.

Licensing Opportunities: Museums and galleries can enter into licensing agreements with game developers, granting them the rights to use their 3D models in commercial game projects. This presents a unique revenue stream for cultural institutions, where they can earn royalties based on the sales and usage of their licensed assets.

Collaborations with Game Studios: Cultural institutions can also explore collaborations with game studios to create virtual experiences or games that incorporate their collections. These collaborations can result in innovative and immersive projects that blend art, history, and gaming, reaching a broader audience and generating new revenue streams.

Integration with Virtual Reality (VR): With the rise of virtual reality, the demand for realistic and immersive experiences has increased. Photogrammetry-generated 3D models can be integrated into VR applications and games, offering users the opportunity to explore virtual museums and interact with digital replicas of historical artifacts and artworks. This can open up partnerships with VR developers and platforms, creating new avenues for revenue generation.

Selling 3D models on game marketplaces not only provides financial benefits but also promotes the cultural heritage represented by the objects. By incorporating these models into video games and virtual experiences, museums and galleries can engage younger audiences and introduce them to art and history in an interactive and entertaining manner.

Process

The technical pipeline of photogrammetry typically involves several steps:

Image Acquisition:

Capture: The first step in photogrammetry is to capture a series of photographs of the object or scene from different angles. These images should cover the entire object or scene from various viewpoints, ensuring proper overlap between images.

Image Preprocessing and Orientation: Before the images can be used for photogrammetry, they often require preprocessing to enhance their quality and remove any distortions or artefacts. This may involve tasks such as lens correction, noise reduction, colour balancing, and image alignment.

Processing (Automated in software):

Feature Extraction: Once the images are properly oriented, distinctive features or keypoints are identi- fied in each image. These features can be corners, edges, or other identifiable points that can be reliably tracked across multiple images. The extraction of features is important for establishing correspondences between images.

Image Matching: In this step, the extracted features from different images are matched to find correspondences. The goal is to identify which key points in one image correspond to the same physical point or feature in another image. These matches are used to establish the relationships between different images and further refine the 3D reconstruction.

Dense Point Cloud Generation: Using the correspondences established in the previous step, a dense point cloud is generated. This point cloud represents a set of 3D points in the scene, with each point having corresponding 2D image coordinates in multiple images. There are various algorithms and techniques for generating dense point clouds, such as multi-view stereo algorithms.

Surface Reconstruction: Once the dense point cloud is generated, a surface reconstruction algorithm is applied to create a continuous 3D surface model. This can be done using techniques like triangulation or volumetric reconstruction. The result is a digital representation of the object’s surface, often represented as a triangular mesh.

Texture Mapping: To enhance the visual appearance of the 3D model, texture mapping is performed. The original input images are projected onto the reconstructed surface, assigning colour information to the corresponding 3D points. This process adds realistic texture and details to the model.

Processing (Manual):

Model Refinement: After the initial reconstruction, various refinement techniques can be applied to improve the accuracy and quality of the 3D model. This may involve processes like noise reduction, hole filling, mesh smoothing, or texture optimization.

Output: The final output of the photogrammetry pipeline is a complete and accurate 3D model of the object or scene. This model can be exported in various file formats, such as OBJ, FBX, or USDz, and can be used for visualisation, analysis, virtual reality, or other applications.

It’s important to note that the specific techniques and algorithms used in each step may vary depending on the software or photogrammetry pipeline being employed. Additionally, the complexity and accuracy of the pipeline can vary depending on the desired output and the quality of the input images.

File formatting

Using the USDz file system, we are able to attach corresponding metadata to each object. This enables us to curate a more holistic narrative around the object, centring craft and creation to bring the object alive. When packaged as USDz, the metadata becomes available to users as they explore the object. This metadata can include:

Collection, storage, conservation and museum item referencing (via MODES)
Artist & craft record – designer, maker, team, process
Live performance media & supporting media (process video, BTS, audio, photography) * Curatorial, EDJI & relevant social context tags

USE CASE EXAMPLES RESOURCES OUTCOMES & REQUIREMENTS
Artistic & Creative: Hybrid Productions Digitally Originated projects Technology partnerships Community creative & Placemaking	Transposing physical props, costume, set from stage to digital platforms AR/mapped overlay or accompanying standalone digital product Game, VR/AR products, installation, online. Outreach projects led by CPPP team	* Scaled Lightbox * Camera rig * Capture team (in house) * Post processing (in house) * Application development or device (commission or partnership) inc UX to access metadata Access to objects post build, pre stage production use (2-6 hrs per object)	Shared aesthetic, look and feel between live perfor- mance and digital offer. Potential of digital overlay during performance Use within production video to map / augment physical objects Extends audience engage- ment in performance beyond the stage. Transposing RSC physical craft skills into our digital products, including design, technical build/making.
RSC Collection	3D capture of objects as they enter the collection Enabling digital loans to other museum/institutions Digital preservation of items otherwise unable to be inducted into collection	* Rig install at Arden Street * Training of collections teams to capture * Post processing pipeline & clean up (in house) * Collation of metadata (DD Archive Administrator) * Licence review – including IP, rights (redraft- ing of collection ownership model & contracts)	Enabling use of delicate objects, otherwise unable to leave the collection stores Opening access to the collection, sharing our assets with other museums, partners and academic institutions
RSC commercial & Brand	Licensing sets of objects and skins/wearables as inventory for third party platforms (such as in game, in metaverse) Minting of NFT versions of collec- tion objects for sale.	* License agreement with owners of objects, poten- tial renegotiation with designers for commercial use * Commercial partnership with platform owners / developers * Legal framework for use & to negotiate terms/in- come	Brand reach to game/digital audiences Marketing activations for major titles/commercial products Income generation (via marketplaces) Individual Giving (HNW individuals)

What are creative captions

Artist Ben Glover introduces Creative Captioning

Intro Ben

Ben is a deaf artist, video designer, motion graphics and virtual reality creator. He’s achieved an MA in digital theatre and BSc in Creative Computing.

Ben’s work carefully combines his interdisciplinary skills, understanding of access and lived experience to produce creative and integrated forms of video design and creative captioning. Working at the forefront of technology and being experienced both on and off stage has allowed him to deliver innovative approaches to his work, particularly in creative captioning. He believes that the best form of access is one that is integrated and included in the creative process from the very beginning.

Highlights of Ben’s work include creative captions for Antony and Cleopatra (Shakespeare Globe), Lived Fiction (Stopgap), The Paradis Files (Graeae Theatre), The Promise (Deafinitely Theatre) and video designs for Best of Enemies (Noël Coward Theatre), Tubular Bells 50th Anniversary Concert Tour and Run Rebel (Pilot Theatre).

Ben was featured in this article by The Guardian (1 July 22) as one of three creatives ‘opening theatres up to disabled audiences’

The challenge

The process of creating captions in this way is labour intensive – with shows often needing several thousand captions designed either through animation, images or text cues. As such, it is an expensive endeavour for productions to build in an accessible tool such Creative Captioning. It also lacks flexibility, which is a barrier for directors and companies refining their show in the crunch of tech & previews.

There isn’t a unified workflow for designers with some utilising surtitling applications such as Glypheo, these solutions limit both creativity and access as they function with “slide-by-slide” captions that impact reading times, particularly with short lines, instead of the scrolling method used as an industry standard by Stagetext. None of these processes take into consideration the text editing process whereby a script is formatted into a captioned version, with appropriate line breaks, name labels, sound descriptions and more.

The solution would be a new software approach that makes the process more agile – using a cue based system and auto video generator to enable captioners and more video designers access to the tool.

Specific guidelines and requirements for the software will be informed by a unified set of guidelines and rules for creative captioning, driven by an association of captioners, designers and access coordinators. This will follow other industry practices such as subtitling guidelines produced by various organisations that will help one of the main challenges in creative captioning which is maintaining a strong level of access in the work whilst being aesthetically creative.

In addition to these, there is also missing research to support audience’s viewing habits in the theatre. A recent study by Ideal Insight showed that 85% of Netflix users in the UK had subtitles turned on, which shows evidence that subtitles are not only for those who are deaf and hard of hearing. An updated study should be conducted in the theatre to correctly analyse the usage of captions and creative captions for all audiences.

Research questions

Test the effectiveness of creative captions with a live audience with immersive technologies (eye tracking, surveys, focus groups)
Discover best working practices and technical workflows for streamlining creative caption design and programming.
Develop industry guidelines on creative captions (best practices, rules and processes) with a contributing board.

The process

During a Fellowship with the RSC, Ben explored how designers and directors use creative captioning in their work and gained feedback on their process as well as the audience’s response.

Ben examined the following three areas:

Planning

Working with the creative team to decide on the aesthetic of the captions and where they fit into the set.

Design

Longest part of the process, making the content. 1,500 lines of caption can take weeks of work.

Control

How content is played back. Manually is currently the most popular method. There are existing tools for captions to be used automatically, but these are not currently creative.

He has also explored how the current tools which are used for designing creative captions will help him investigate what they are capable of and what limitations they have. He consulted with colleagues at the RSC to understand each production and requirements, and the Company’s current offer – for example how captions can only be seen from some seats and are not used for all performances.

Findings

Ben has developed a paper prototype for a creative captioning software solution informed by his own and other creative captioners working process. The next stages would involve:

Software development of a functional version.
Testing in an R&D phase.
Continued development working alongside a full creative captioned production.
Launch and release of the software with maintained development and support.

A software partner should be identified to support Ben in developing a working prototype that can be tested alongside his work and understand the essential tools needed such as; cloud collaboration, real-time text animation and templating workflows. The RSC will support Ben with an R&D to utilise the software prototype in a working environment with a writer, director and actors.

New research with audience use of captions will enable a better understanding of audiences needs and analyse the impact of captions. This will be conducted with both an open captioned performance and a creative captioned performance to help compare the two approaches and set a benchmark for identifying trends with audience engagement.

Ben has recently completed both a DYCP and Community Fund project with CRIPtic Arts, both of which explored two areas of creative captioning. His DYCP was focused on Creative Captioning and Access in which he was mentored by various captioners to understand their working practice from an access standpoint. He has integrated this understanding into his own creative practice and explored various methods for achieving these with unofficial methods, such as; adding scrolling text captions to Qlab, editing scripts for readability and word count, adding sound descriptions to captions. With CRIPtic Arts, he conducted research across various organisations, particularly in small-scale theatre, to explore the main barriers to providing captions including budgets, lack of awareness and technology. The key findings were that a lack of a study for the effectiveness of captions meant many organisations felt captions weren’t necessary and standard approaches with organisations such as Stagetext were too costly. The outcome of this research was to develop an online resource containing guidance and information on captioning with key information for organisations and captioners, led by a governance structure.

What is AI ?

Artificial Intelligence (AI) has made significant strides in recent years, radically transforming various aspects of society. In its current form, AI encompasses a range of technologies and techniques, including

machine learning
natural language processing
computer vision and
deep learning

These technologies have found applications across industries such as healthcare, finance,transportation, and entertainment and will, over the next 10 years, radically transform oursociety.

There are considerable concerns with the impact AI will have on humanity. The models /data sets they work off are inherently bias due to their ownership and design by a handfulof heavily resourced, largely unregulated tech companies.

Impact of AI on Storytelling and Creative Artforms such as theatre:

Artistic Creation

AI will be the largest enabler of written content, music, and visual art this decade. Forexample, AI-powered tools can write creative works including full plays, create‘photography’ and visual art, compose music from prompts – visual models such as DALLE and MidJourney are soon to be capable of creating not just imagery, but full 3D worlds forplatforms like video games and virtual production. Whereas AI is currently easy to spot when working creatively, it is widely used to assist the creation process – fundamentally changing the process by which our artists author works.

Digital Media & Content

Filmmakers and video game developers are using AI to enhance special effects, animation, and virtual worlds. AI-driven tools can generate realistic facial expressions, landscapes, and characters, reducing the time and cost of production by reducing artistic input from writers, directors and actors.

Cultural Consumption

Streaming platforms and social media leverage AI algorithms to curate and recommend content to users. This impacts how people discover and engage with creative works, reshaping the entertainment industry.

Process

As part of the 2022 RSC Rothschild Digital Fellowships, theatre maker Kirsty Housley set out to explore how women’s voices are policed through technology and digital tools. Her work investigates the bias’s inherent in the sector and how that bias impacts the creation of new technologies.

Kirsty explored how AI biases in voice assistants, search engines and LLM’s seemed to be the latest instalment in a continuum of women’s voices being discredited throughout human history. Taking inspiration from Cassandra, Rachel Carson (whose extraordinary book Silent Spring was discredited by her male counterparts who repeatedly called her a hysterical housewife), Brooksley Born, Anita Hill and a thousand other women who weren’t believed and should have been, Kirsty looked at how history is repeating itself through the technology we now use, and the ways in which we might instead use it to break these cycles.

Kirsty focused on where AI is heading through her Fellowship. She looked at gender and voice both physically and politically to explore voice modification, audience interaction with voice AI and dig more into Rachel’s story. She also explored how audiences might like to interact with this technology and how artists could combine it with storytelling.

Collaborations

Kirsty put out a call to sound artists and composers to collaborate, and this resulted in her working with Ivor Novello Award winning composer Sarah Angliss. Sarah’s music explores the sonorities of voices and ancient instruments, revealing and augmenting them with her distinctive electronic techniques. Together they experimented to push the boundaries of sound, technology and voice.

In her presentation Kirsty used an actor to present her work experimenting with words from Anita Hill, Christine Ford, and Greta Thunberg to create new sentences, with the same things said but in different ways.

Sarah Angliss, composer and sound artist, continued to explore Respeecher beyond the fellowship in her own practice. The video below is an experiment with voice cloning – rendered from a single male voice reading Lady Macbeth’s Unsex Me Here soliloquy.

Sarah used MaxMSP, Processing and Respeecher as tools here.

Inspiration & tools

Holly+ is Holly Herndon’s digital twin. The custom voice instrument and website by Never Before Heard Sounds allows for anyone to upload polyphonic audio and receive a download of that music sung back in Herndon’s distinctive voice. In the fast approaching era of deepfakes, Herndon embraces the utility of the near future of this technology rather than be disempowered by it. This creative and economic experiment proposes a way for artists to take control of their digital selves without obstructing experimentation with punitive copyright lawsuits.

“I love Holly Herndon. Here she’s cloned herself vocally and is allowing anyone to use her voice, via Holly +, to create art. Holly + is also a DAO (decentralised autonomous organisation). Holly + was made in collaboration with Never Before Heard sounds, so I had a meeting with them and talked about how their pre existing software might be used by us to create a live piece of work, using the voices of the women who are in the room on the day. You’d never experience the same soundtrack twice.” KIRSTY HOUSLEY

Respeecher is an AI-powered Voice Marketplace that offers voice cloning services for content creators, including filmmakers, game creators, voice actors, and YouTubers. With Respeecher, users can speak in another person’s voice while preserving emotions, volumes, and emphasis, enabling them to create unique and engaging content.

Key Features of 5G:

Increased Speeds
- 5G offers speeds that are 10 to 100 times faster than 4G, potentially reaching gigabit-per-second levels.
Lower Latency
- While 4G latency is around 30-50 milliseconds, 5G promises to reduce this to as low as 1 millisecond, which is crucial for real-time applications.
Greater Capacity
- 5G can support a larger number of connected devices. If, for example, we wanted to deliver Realtime data to 1000 people in the auditorium.
Improved Reliability
- 5G offers more stable and reliable connections, as reliable as cabled data transfer
Network Slicing
- This allows service providers to segment the network to provide tailored services, the potential for our own bandwidth that is independent of external demand

A schematic showing the progress from 1G to 5G, from the 1980s-2020.

Potential uses for 5G in theatre & live performance

5G, with its high data transfer rates, low latency, and potential for massive device connectivity, can significantly enhance the experience of live events for both organizers and attendees. Here are some potential use cases of 5G in live events:

Enhanced Live Streaming

With faster and more reliable connections, live streaming of events can be done in higher resolutions like 4K or even 8K, providing attendees and remote viewers with a clearer and smoother experience.

Augmented Reality (AR) and Virtual Reality (VR) Experiences

Low latency and high bandwidth allow for real-time AR and VR experiences. Attendees can use AR glasses to see additional information about the event, or remote attendees can experience the event virtually using VR headsets.

Real-time Analytics and Feedback

Event organizers can gather real-time feedback from sensors, wearables, or mobile apps, allowing them to make instant changes to improve the audience experience. Needed for delivering data to (for eg) an AR headset based on where the user is looking.

Instant Content Sharing

Ability for audiences to feedback and effect Realtime, multiuser game engine experiences.

Interactive Displays and Kiosks

These can be updated in real-time with relevant information, advertisements, or even user-generated content.

Wearable Tech

Attendees can use wearables, like smartwatches or AR glasses, which can enhance their event experience by providing real-time information, navigation, artistic or even translation services (inc access provision).

Incorporating 5G into live events can create a more immersive, interactive, and efficient experience for everyone involved. As the technology continues to evolve, these areas are yet to be explored but have the potential to offer new forms of hybrid theatre for remote and in person audiences.

Research Questions

Can the lighting control system hold a stable session over 5G? [MANet3]
Can real fixtures receive streaming data and synchronise output at two locations? [Artnet]
Can the lighting control system receive valid 3D positional data from a remote location? [PosiStageNet]
Can one image be sent to two locations and arrive somewhat synchronised and without artefacts? [NDI]
How fluid does the NDI KVM functionality feel? [NDI]
Does the latency of a network time clock remain constant over time and after re-starts? [SMPTE 2110]
Can audio and video be sent and received with a latency that is tolerable circa 25ms? [DANTE]
More for use on local 5G? [XDIP KVM]
Can the Disguise media servers hold a network session? [D3 Media Server]

Process & technical

RSC partnered with Ericsson’s 5G team to explore sector specific use cases for theatre and live performance. The technology has the potential to enable data transfer between show control and fixtures – removing the need for data cabling both in auditorium and on location. For example, if we wanted to sail a boat down the Avon with integrated projections, light and audio units positioned on the boat.

The Research took place at Ericsson’s 5G Test Bed at their Coventry lab.
We created three zones – A as the origin, B as remote location 1 and C for remote location 2 for additional protocol testing.

Protocols:

MANet3	In-built network monitoring tools
Artnet	High frame rate video camera pointing at real lighting fixtures.
Streaming ACN – (As Artnet)	PosiStageNet – Screen recording.
SMPTE 2110	High frame rate video camera.
NDI	High frame rate video camera.
DANTE	Split channel audio recording via M4 Audio interface for audiogram measurement (Audition).
XDIP KVM	High frame rate video camera.
D3 Media Server	In-built network monitoring tools

5g is being delivered in two phases – Non-Standalone and Standalone. Non-Standalone uses the 4g network and is effectively a hybrid of 4G & 5G (what’s currently available on consumer mobile). Standalone is native 5G, using new network services to deliver the full benefits – critically it enables End to End Network Slicing, dedicated bandwidth for our own applications, providing reliability needed for theatre production applications.

In summary, while 5G technology as a whole promises transformative changes in the realm of wireless communication, the Standalone architecture represents the purest, most advanced form of this technology.

Outcomes and next steps

USE CASE #1:

Using a 5G dedicated network to manage data transfer between show control to fixtures and units. Mitigates the requirements for ethernet and other data cables in our auditoriums. Offers more flexibility for rig positions, mobile/moving units, dynamic automations.

EXAMPLE: Show control able to send and receive data from fixtures without data cables – streaming video to projectors, LX and sound.
Prop embedded battery powered lighting fixtures controlled via lighting desk, rather than actor or SM operated during show.

RESOURCES & REQUIREMENTS: Further R&D to test design infrastructure, actual latency stability and development of back up & fall back options, involving technical & production teams
Development of 5G node to add network functionality to non-smart units Commission of a 5G standalone Network Slice, business case for subscription charges

USE CASE #2:

Using 5G standalone as a temporary solution that enables site specific / location-based show delivery

EXAMPLE: A performance on the Dell, with scenes on the Avon enabled with mobile technical fixtures – LX and Sound on boats, in trees, via camera feeds – all without the need of cabling

RESOURCES & REQUIREMENTS: Temp dedicated 5G standalone network license for the length of the project, likely a corporate partnership with Ericsson 5G
Battery solutions for all fixtures

USE CASE #3:

Immersive production utilising XR headsets with 5G delivering data to each headset, without the need for local processing (and therefore lighter headsets). Ability to run shared experience for 000’s of simultaneous users.

EXAMPLE: Production with layered digital element that is responsive to the live performance and enables individual audience members to affect the world. Regardless of where you sit, the graphics will overlay the live show, giving a personal view of hybrid production. Imagine – presenting an animated character as part of the company; Bottom becomes the ass; a storm fills the auditorium.

RESOURCES & REQUIREMENTS: Hardware dependant, technology not currently feasible to deliver at scale.

Heading needed

Investigator: Adam Clarke (as part of 2022 Rothschild Digital Fellowships)

Adam is well known for his groundbreaking work in Minecraft and was one of the first creators to bring digital games into museums in collaboration with Tate Britain. He’s also used video games to augment a global audience to explore a local community art project Temple craft with the artichoke trust and directed the Minecraft elements of Play-Craft live, that combine live theatre stage show with a video game and a live-streamed audience.

The Fellowship has provided Adam with the opportunity to experiment, exploring how we can combine immersive video game-space, theatre space, and Livestream audiences and how to bring these three diverse audiences together to build a narrative toolset that helps transition stagecraft to immersive game environments.

Although already established in his field, the Fellowship enabled Adam to think differently about his creative practice and think about using animation in Minecraft. He described it as a ‘massive opportunity to play, explore and experiment’.

Why Minecraft for Educational Storytelling?

Emotional Connection

Minecraft allows players to connect emotionally with the environment and characters. Its is an embodied experience within the world of the map – be that a forest, castle or other setting. The emotional connection enhances the retention of the educational material.

Contextual Learning

The game offers a lifelike context where abstract concepts are made tangible. Players aren’t just reading about ideas and concepts but exploring them in a realistically modelled world.

Engagement and Attention

With its interactive nature, Minecraft captures and sustains attention. The varied quests in game serve as mini-narratives, keeping players continuously engaged and invested in their learning journey.

Encourages Exploration

Minecraft’s open-world environment encourages critical thinking and exploration. For example, challenges invite players to delve deeper into topics, allowing time to explore nuance and understand motive.

Relevance and Application

Playing roles and characters in game can provide an accessible first introduction to new environments and detailed characters like we see in Shakespeares plays. Young people explore on their own terms which informs their learning in the classroom.

Simplification of Complexity

Minecraft serves as a scaffold where complex themes can be broken down into understandable narratives and activities.

Reinforcement and Repetition

The game’s story arcs and quest lines offer natural opportunities to repeat key educational concepts, reinforcing learning objectives.

Alternative platforms

Alongside Minecraft, digital products like Roblox and Fortnite are increasingly turning their attention to providing educational experiences. Between them, they hold the focus of hundreds of millions of young people. Whilst in battle modes, the chaotic multi user environments are not suitable for creative projects like ours, almost 50% of user time in these games are spent in creative mode, space where you can turn off functions like weapons & killing games.

Examples:

* Fortnite’s March Through Time, exploring the life of Martin Luther King (Aug 21) & Voices of the Forgotten (commemorating the Holocaust)

* Roblox tie ins with Met Museum of Art, Parsons School of Design and US Careers Center (where you can apply for a real world job)

Process

Telling epic stories within Minecraft

Adam used the RSC’s in-house motion capture expertise to develop Vlox character building and make their movement more fluid – to be able to represent characters as complex as those found in Shakespeare’s plays. He used this to build two projects – Hamlet’s castle and a characterised version of Macbeth in a tower block in Leamington Spa (see a short video Adam has uploaded to YouTube below).