Getting Started with Transmodal Analysis (TMA) in Python¶

The Python port of the R tma package's getting-started vignette. qe-tma (import tma) has the same rules(), contexts(), context_tensor() and accumulate() as the R package and gives the same numbers.

Introduction¶

Transmodal Analysis (TMA) is a framework for analyzing multimodal data in Quantitative Ethnography (QE). It extends state-dependent models such as Epistemic Network Analysis (ENA) to data from several concurrent streams: verbal communication, tool use, eye-tracking events, sensor logs.

These streams work on different time scales and involve different actors:

  • Chat messages may stay relevant for several minutes, for every team member.
  • Action or resource logs may matter only for a moment, and only to the user who made them.
  • Some actions are seen only within a sub-group; others are seen by everyone.

TMA models these relationships with:

  1. Horizons of Observation (HOO): rules for which events each unit can see, across actors and modalities.
  2. Context tensors: a time window and a connection weight for each modality.
  3. Fast accumulation: libtma (C++, compiled into qe-tma) accumulates the connection matrices.

For the methodology, see:

Shaffer, D. W., Wang, Z., & Ruis, A. R. (2025). "Transmodal Analysis." Journal of Learning Analytics. doi:10.18608/jla.2025.8423

Installation¶

pip install qe-tma --extra-index-url https://qe-libs.org/py/simple/

This walkthrough needs qe-tma 0.1.11 or later (accumulate(hoo_rules=) and tma.datasets). Until 0.1.11 is released, install the development build:

pip install --pre qe-tma --extra-index-url https://qe-libs.org/py/dev/simple/

Walkthrough: multimodal data accumulation¶

This walkthrough uses the sample data shipped with the package, the same test_mockdata as R's data("test_mockdata", package = "tma").

1. Load the package and inspect the data¶

In [1]:
import pandas as pd
from tma import accumulate, context_tensor, contexts, rules
from tma.datasets import load_test_mockdata

test_mockdata = load_test_mockdata()

# Keep one chat group for the demonstration
sample_df = test_mockdata[test_mockdata["chatGroup"] == "PAM"].reset_index(drop=True)

# The first 6 multimodal events
sample_df[["userID", "condition", "modality", "timeStamp", "A", "B", "C"]].head(6)
Out[1]:
userID condition modality timeStamp A B C
0 User1 FirstHalf chat 1379410965 1 0 0
1 User2 FirstHalf resource 1379411071 0 1 1
2 User3 FirstHalf chat 1379411300 1 0 0
3 User1 FirstHalf resource 1379411494 0 0 1
4 User2 FirstHalf chat 1379411667 0 1 0
5 User3 FirstHalf chat 1379411813 1 0 1

The columns used here:

  • userID and condition: the participant and the experimental condition.
  • modality: the kind of event, "chat" (communication) or "resource" (interaction with a resource).
  • timeStamp: epoch time in seconds.
  • A, B, C: binary codes for cognitive or behavioral categories.

2. How horizons and tensors work¶

Here is how TMA processes the 6 rows in the table above.

A. The global event stream¶

The six events arrive in order across participants and modalities:

Row:           [R1]          [R2]          [R3]          [R4]          [R5]          [R6]
User:         User1         User2         User3         User1         User2         User3
Modality:      chat       resource        chat        resource        chat          chat
Time (s):   1379410965    1379411071    1379411300    1379411494    1379411667    1379411813
Codes:         [A]         [B, C]          [A]           [C]           [B]         [A, C]

B. Filtering by Horizon of Observation (HOO)¶

This analysis uses two rules:

  • chat messages in group PAM are public to everyone in group PAM.
  • resource interactions are private to the user who clicked them.

TMA splits the global stream into a separate context horizon for each participant. Here are User 1's and User 2's:

┌──────────────────────────────────────────────────────────────────┐
│ Global Stream: [R1: User1 Chat] → [R2: User2 Res] → [R3: ...]    │
└────────────────────────────────┬─────────────────────────────────┘
                                 │
        ┌────────────────────────┴────────────────────────┐
        ▼                                                 ▼
┌───────────────────────────────┐ ┌───────────────────────────────┐
│ User 1 Context Horizon        │ │ User 2 Context Horizon        │
│ • R1: User 1 Chat [A]         │ │ • R1: User 1 Chat [A]         │
│ • R3: User 3 Chat [A]         │ │ • R2: User 2 Resource [B, C]  │
│ • R4: User 1 Resource [C]     │ │ • R3: User 3 Chat [A]         │
│ • R5: User 2 Chat [B]         │ │ • R5: User 2 Chat [B]         │
│ • R6: User 3 Chat [A, C]      │ │ • R6: User 3 Chat [A, C]      │
│                               │ │                               │
│ ❌ Excluded:                  │ │ ❌ Excluded:                  │
│   R2 (User 2 Resource click)  │ │   R4 (User 1 Resource click)  │
└───────────────────────────────┘ └───────────────────────────────┘

C. Context tensor windows and weights¶

For each event (the response $r$), TMA looks back through that user's context horizon. It:

  1. Finds the earlier events within the modality's time window ($w_m$). These form the ground $g$.
  2. Applies the modality weight ($W_m$) to set the connection strength.
  3. Computes the co-occurrence cross-product ($g \otimes r$).

Example: row 4 as the response

  • User 1 produces row 4: a resource click at $t = 1379411494$ with code [C].
  • Looking back through User 1's horizon:
    • Row 3 (User 3 chat at $t = 1379411300$, code [A]): $\Delta t = 194$ s. The chat window is $w_{\text{chat}} = 360$ s, and $194 \le 360$, so row 3 is in the ground.
    • Row 1 (User 1 chat at $t = 1379410965$, code [A]): $\Delta t = 529$ s. Since $529 > 360$, row 1 has left the window.
  • Result: User 1 accumulates a directed connection from A (chat) to C (resource).

3. Define the units of analysis¶

Units of analysis are the entities that each get their own connection network.

In [2]:
unit_cols = ["userID", "condition"]
codes = ["A", "B", "C"]

4. Write the Horizon of Observation (HOO) rules¶

R's rules() takes expressions that use UNIT for the focal unit. In Python, each rule is a function rule(df, unit): df is the data, unit is a row of the focal unit (R's UNIT), and the function returns a boolean Series. A row is visible to a unit if it matches any rule.

In [3]:
hoo_rules = rules(
    # Chats: visible to everyone in the same chat group and condition
    lambda df, unit: (df["modality"] == "chat")
                     & (df["chatGroup"] == unit["chatGroup"])
                     & (df["condition"] == unit["condition"]),
    # Resource clicks: visible only to the user who made them
    lambda df, unit: (df["modality"] == "resource")
                     & (df["userID"] == unit["userID"])
                     & (df["condition"] == unit["condition"]),
)

For the common case of "rows that share the unit's values in these columns", conversation_rules("chatGroup", "condition") builds the rule for you.

5. Build the context horizons¶

contexts() applies the rules to find the rows each unit can see:

In [4]:
context_model = contexts(sample_df, units=unit_cols, hoo_rules=hoo_rules)
context_model
Out[4]:
<ContextModel 3 units, 12 rows>

Check which rows are in User 1's horizon: the group's chats and User 1's own resource clicks, but not User 2's clicks.

In [5]:
cols = ["userID", "modality", "timeStamp", "A", "B", "C"]
user1_rows = context_model.contexts_map["User1::FirstHalf"]
context_model.data.loc[user1_rows, cols].head(6)
Out[5]:
userID modality timeStamp A B C
0 User1 chat 1379410965 1 0 0
2 User3 chat 1379411300 1 0 0
3 User1 resource 1379411494 0 0 1
4 User2 chat 1379411667 0 1 0
5 User3 chat 1379411813 1 0 1
6 User1 resource 1379500979 1 0 0

6. Set up the context tensor (windows and weights)¶

The context tensor sets how far back in time each modality can connect (the window) and how much each modality counts (the weight).

The tensor is a NumPy array. With only a mode column, its shape is (n_modes, 2). The last axis is [weight, window] (index 0 is the weight, index 1 the window), and tensor.mode_levels gives the order of the modes. So R's tensor["chat", "window"] is tensor[chat, 1] in Python.

In [6]:
# One tensor entry per modality; everything starts at window 0, weight 0
tensor = context_tensor(
    sample_df,
    sender_cols=None,
    receiver_cols=None,
    mode_col="modality",
    default_window=0,
    default_weight=0,
)
chat = tensor.mode_levels.index("chat")
resource = tensor.mode_levels.index("resource")
WEIGHT, WINDOW = 0, 1

# Time windows, in seconds
tensor[chat, WINDOW] = 360       # chat messages stay active for 360 seconds
tensor[resource, WINDOW] = 180   # resource interactions stay active for 180 seconds

# Modality weights
tensor[chat, WEIGHT] = 1.0       # chat weight = 1.0
tensor[resource, WEIGHT] = 2.0   # resource interactions count double

pd.DataFrame(tensor.array, index=tensor.mode_levels, columns=["weight", "window"])
Out[6]:
weight window
chat 1.0 360.0
resource 2.0 180.0

7. Accumulate the network connections¶

accumulate() builds each unit's context from hoo_rules (as contexts() did above) and accumulates its connections:

In [7]:
accum_result = accumulate(
    sample_df,
    units=unit_cols,
    codes=codes,
    hoo_rules=hoo_rules,
    tensor=tensor,
    time_col="timeStamp",
    binary=True,
    ordered=True,
)
accum_result
Out[7]:
<TMAAccumulation 3 units x 9 connections (ordered)>

The result is a TMAAccumulation. Its fields follow R's accumulate():

Python R What
connection_counts_ connection.counts Accumulated connection strengths: units × connections
meta meta.data The metadata for each unit
unit_labels_ ENA_UNIT Unit labels ("User1::FirstHalf", ...)
connection_names_ column names "ground & response" for each connection
self_connection_counts_, miss_counts_ self.connection.counts, miss.counts Self-connections and missed co-occurrences
row_connection_counts_ model$row.connection.counts The same, per data row

What the miss and self-connection arrays count is in Reading Accumulation Results.

Here are the accumulated co-occurrences as a table. "A & C" is ground A → response C. User 1's A & C is the row-4 connection from section 2C, at the resource weight of 2:

In [8]:
pd.DataFrame(
    accum_result.connection_counts_,
    index=accum_result.unit_labels_,
    columns=accum_result.connection_names_,
)
Out[8]:
A & A B & A C & A A & B B & B C & B A & C B & C C & C
User1::FirstHalf 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0
User2::FirstHalf 0.0 0.0 0.0 2.0 0.0 2.0 1.0 2.0 0.0
User3::FirstHalf 1.0 2.0 1.0 0.0 0.0 0.0 1.0 2.0 0.0

Directed vs. undirected accumulation¶

TMA supports both directed (ordered) and undirected (unordered) accumulation through ordered:

  • Directed (ordered=True): tells $A \rightarrow B$ apart from $B \rightarrow A$ by time order (ground → response). Gives $n^2$ connection columns for $n$ codes.
  • Undirected (ordered=False, the default): folds each pair of directed co-occurrences into one symmetric pair ($A \leftrightarrow B$). Gives $\binom{n}{2}$ connection columns.
In [9]:
accum_undirected = accumulate(
    sample_df,
    units=unit_cols,
    codes=codes,
    hoo_rules=hoo_rules,
    tensor=tensor,
    time_col="timeStamp",
    binary=True,
    ordered=False,
)

# Column names for the undirected connection pairs
accum_undirected.connection_names_
Out[9]:
['A & B', 'A & C', 'B & C']

Next steps¶

qe-ena models a TMA accumulation the way rENA does in R: normalization, rotation (SVD or means rotation) and node positions. It needs qe-ena 0.1.10 or later (until that is released, the development build: pip install --pre qe-ena --extra-index-url https://qe-libs.org/py/dev/simple/).

from ena import ENA

model = ENA().fit(accum_result)
model.points_       # each unit's position in the projected space

The low-level libtma kernels (apply_tensor_unit, accumulate_stanza, ...) are in tma.libtma. The API reference is at qe-libs.org/py/docs/qe-tma.