Finite State Machine

FSM is the brain of the 3DAC system, governing the entire chat lifecycle and ensuring consistent behavior across all system states. It defines a series of states and makes state transitions throughout the system. The FSM uses a condition-based event queue to process external signals such as user interactions, network events, which consists of 2 stages:

• System Startup: Ensures that all static and runtime assets are available before entering the chat, including environment model, character mesh, local animations(e.g., audio/motion/face for apologize behavior). It also generates an opening remark as the avatar’s first sentence.

• Interactive Conversation: Triggered when the user enters the chat, the system orchestrates stream lifecycle management, synchronizes animation playback with the runtime pipeline, and applies robust error handling and recovery to deliver a seamless chat experience.

The following figure illustrates the overall workflow of FSM:

FSM

States

INIT

• Initialize the FSM system.

• Transit to:

  • WAITING_FOR_FRONTEND_READY when initialization finished.

WAITING_FOR_FRONTEND_READY

• Wait for Babylon scene to be ready.

• Transit to:

  • SPAWN_ENVIRONMENT when received FRONTEND_READY condition.

  • EXIT when scene startup timed out.

SPAWN_ENVIRONMENT

• Load HDR environment and ground model using user settings.

• Transit to:

  • SPAWN_CHARACTER after scene setup succeeds.

  • EXIT when environment setup fails.

SPAWN_CHARACTER

• Loads character configuration and assets.

• Transit to:

  • WAITING_FOR_ALGORITHM_READY_ON_START when the character is successfully added to the scene.

  • EXIT when configuration fetch or model loading fails.

WAITING_FOR_ALGORITHM_READY_ON_START

• Health-check for algorithm services and validate stream buffer.

• Transit to:

  • CHECK_AND_UPDATE_ASSETS when health-check and stream evaluation passed.

  • ALGORITHM_GENERATION_FAILED when health-check or stream evaluation failed.

CHECK_AND_UPDATE_ASSETS

• Sync character local motion/audio/face into runtime and generate opening remark.

• Transit to:

  • WAITING_FOR_USER_START_GAME when synchronization and generation finished.

  • EXIT if any error occurs.

WAITING_FOR_USER_START_GAME

• Wait for user to start chat and accumulate opening remark stream.

• Transit to:

  • ACTOR_ANIMATION_STREAMING when received USER_START_GAME condition and the opening remark is still streaming.

  • WAITING_FOR_ACTOR_ANIMATION_FINISHED when received USER_START_GAME condition and the opening remark is fully received.

  • EXIT when microphone check fails or opening remark data is missing.

IDLE

• Default rest state, which listens for the user input.

• Transit to:

  • WAITING_FOR_USER_STOP_RECORDING when received USER_START_RECORDING condition.

WAITING_FOR_USER_STOP_RECORDING

• The user is uploading audio.

• Transit to:

  • WAITING_FOR_LOCAL_ANIMATION_INTERRUPTED when received USER_STOP_RECORDING condition.

WAITING_FOR_LOCAL_ANIMATION_INTERRUPTED

• Wait for local animation to interrupt.

• Transit to:

  • WAITING_FOR_ACTOR_RESPOND_GENERATION_FINISHED when received ANIMATION_FINISHED signal.

WAITING_FOR_ACTOR_RESPOND_GENERATION_FINISHED

• Wait for algorithm response generation.

• Transit to:

  • WAITING_FOR_USER_STOP_RECORDING when received USER_START_RECORDING signal.

  • ACTOR_ANIMATION_STREAMING when the animation started streaming.

  • WAITING_FOR_ACTOR_LEAVING_FINISHED when response type from the algorithm service is leave.

  • ALGORITHM_GENERATION_FAILED when algorithm service failed to return valid stream data.

  • WAITING_FOR_STREAMED_ANIMATION_INTERRUPTED when received USER_INTERRUPT_ANIMATION signal.

ACTOR_ANIMATION_STREAMING

• Receive and feed stream data to runtime. If the stream buffer is exhausted, handle the pause/recover event.

• Transit to:

  • WAITING_FOR_ACTOR_ANIMATION_FINISHED when stream ended or playback finished signals received.

  • WAITING_FOR_STREAMED_ANIMATION_INTERRUPTED when received USER_INTERRUPT_ANIMATION signal.

  • ALGORITHM_GENERATION_FAILED on stream unavailable or unknown errors.

WAITING_FOR_STREAMED_ANIMATION_INTERRUPTED

• Interrupt streamed animation, return to listen state and restart audio recording.

• Transit to:

  • WAITING_FOR_USER_STOP_RECORDING when the interrupt event is sent to runtime and the audio recording started.

WAITING_FOR_ACTOR_ANIMATION_FINISHED

• Wait for character animation to complete and check for relationship and emotion updates.

• Transit to:

  • IDLE when received ANIMATION_FINISHED signal.

  • WAITING_FOR_STREAMED_ANIMATION_INTERRUPTED when received USER_INTERRUPT_ANIMATION signal.

WAITING_FOR_ACTOR_LEAVING_FINISHED

• Wait for leaving animation to complete.

• Transit to:

  • EXIT when received ANIMATION_FINISHED signal.

ALGORITHM_GENERATION_FAILED

• Switch to local error/apology animation.

• Transit to:

  • WAITING_FOR_ACTOR_APOLOGIZE_FINISHED when the switch animation signal is sent to runtime.

WAITING_FOR_ACTOR_APOLOGIZE_FINISHED

• Wait for apology animation to complete.

• Transit to:

  • IDLE when received ANIMATION_FINISHED signal.

EXIT

• Terminate state machine and notify user.

• Transit to:

  • (terminal state)