Adverse Events

How can we explainably identify adverse events?

This application uses a multi-agent LLM workflow to identify whether user feedback from a clinical trial is an adverse event or not.

The workflow has tasks, each running a different model to evaluate the text for potential adverse events. Finally, it summarizes the results in a Pharmacovigilance assessment summary.

Here's a breakdown of the workflow:

Context Analysis: The basic model analyzes the clinical description to determine if there is any mention of an adverse event.
BioClin Analysis: An intermediate model performs a more detailed evaluation, considering clinical reasoning to assess the presence of adverse reactions.
LLM as a Judge Analysis: A specialized model acts as a judge, evaluating the feedback from the previous analyses based on predefined criteria.
Judge Feedback to Context Analysis: The judge provides feedback on the context analysis, which is then used to refine the initial evaluation.
Judge Feedback to BioClin Analysis: Similar to the previous step, the judge reviews the bio-clinical analysis and offers feedback for improvement.
Context Analysis - Revised: The basic model re-evaluates the clinical description, incorporating the judge's feedback to enhance its analysis.
BioClin Analysis - Revised: The intermediate model revisits its analysis, integrating the judge's feedback for a more accurate assessment.
Judge Summary: Finally, the judge summarizes the findings from all analyses, generating a comprehensive report that includes a pharmacovigilance assessment summary in JSON format.

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#BB2528',
      'primaryTextColor': '#fff',
      'primaryBorderColor': '#7C0000',
      'lineColor': '#F8B229',
      'secondaryColor': '#006100',
      'tertiaryColor': '#fff'
    }
  }
}%%

flowchart LR
  A[Context Analysis] --> C[LLM as a Judge Analysis]
  B[BioClin Analysis] --> C
  C --> D[Judge Feedback to Context Analysis]
  C --> E[Judge Feedback to BioClin Analysis]
  D --> F[Context Analysis - Revised]
  E --> G[BioClin Analysis - Revised]
  F --> H[Judge Summary]
  G --> H
  C --> H

Pick a sample clinical trial description

... or enter your clinical trial description

Model Settings

Basic Model

Intermediate Model

LLM Judge Model

Slow down the rendering

Advanced Settings

Basic Prompt

Intermediate Prompt

LLM Judge Prompt

A subject has provided assessment of presence of an adverse event. Evaluate it on the following parameters:

- Temporal Ambiguity and Causality: Determines if there's a clear timeline between drug intake and symptoms. Without a specific temporal link, an adverse event will not be assumed.
- Language Variability and Colloquialisms: Adjusts for non-medical or varied terms describing symptoms (e.g., "feeling woozy" could imply different conditions). The assistant will only confirm an AE when it can interpret symptoms accurately.
- Context Dependencies: Identifies whether symptoms could relate to the condition being treated, unrelated issues, or drug side effects. A conclusion is only drawn if context supports a drug-related cause.
- Multiple Drug Interactions: When multiple medications are mentioned, the assistant evaluates if symptoms could result from interactions rather than single drugs, erring on the side of caution.
- Incomplete Information: If key details are missing (medication names, timing, etc.), it will avoid conclusions and recommend more information.
- Severity Assessment Challenges: Avoids conclusions where severity is unclear from casual descriptions, as phrases like "massive headache" are hard to quantify.
- Reliability and Verification Issues: Uses only reliable information and will avoid conclusions based on second-hand reports or exaggerations.
- Cultural and Regional Variations: Considers cultural differences in symptom descriptions; avoids conclusions when meaning may vary.
- Sarcasm and Figurative Language: Ignores non-literal expressions and does not conclude adverse events from figurative statements.
- Conditional Symptoms: Examines if symptoms occur only under specific conditions (e.g., "only dizzy on an empty stomach") and avoids conclusions unless these patterns are clear.
- Evolving Symptoms: Takes into account symptom progression over time to differentiate adverse events from disease progression or other causes.
- Privacy and Identity Verification: Without clear user verification, it refrains from drawing definite conclusions.
- Natural Disease Progression vs. AE: Differentiates between disease symptoms, side effects, and disease progression to avoid misattributions.
- Reporting Bias: Considers reporting tendencies, recognizing that severe or negative experiences are more likely to be shared online.
- Technical Challenges: Handles issues with spelling, abbreviations, or context-dependent meanings to avoid inaccuracies.

Then decide whether the feedback is an adverse reaction or not. Remember that you should call a medical event an adverse event only if a clear relationship is established between drug intake and medical condition.

Render each step of the reasoning like this:

<details>
<summary>Evaluating: Temporal Ambiguity and Causality</summary>
...
</details>
Result: No temporal link between drug intake and symptoms. No adverse event

<details>
<summary>Evaluating: Language Variability and Colloquialisms</summary>
...
</details>
Result: Interpretation of symptoms is accurate (feeling woozy could be dizziness). Adverse event confirmed

End with one of these:

**Conclusion: Adverse event confirmed**
**Conclusion: Not an adverse event**

But remember that you should conclude that a health condition is an adverse event only if you are able to clearly establish relationship between drug intake and symptoms / health condition. Avoid false positives as much as you can.

LLM Judge Feedback Prompt

Model Revision Prompt

Summary Prompt

Summary Schema

{
  "type": "object",
  "properties": {
    "status": {
      "type": "string",
      "enum": ["VALID CASE", "INVALID CASE", "PENDING REVIEW", "CLOSED"]
    },
    "causality": {
      "type": "string",
      "enum": ["Certain", "Probable", "Possible", "Unlikely", "Conditional", "Unassessable", "Unrelated"]
    },
    "seriousness": {
      "type": "string",
      "enum": ["Serious", "Non-Serious", "Life-threatening", "Hospitalization required", "Disability", "Congenital anomaly", "Death"]
    },
    "expectedness": {
      "type": "string",
      "enum": ["Listed", "Unlisted", "Unknown", "Unexpected"]
    },

"primary_event": {
      "type": "string",
      "description": "The main adverse event category. Examples include 'Gastrointestinal Disorder', 'Cardiac Event', 'Skin Reaction', 'Neurological Disorder', etc."
    },
    "meddra_pt": {
      "type": "string",
      "description": "The MedDRA (Medical Dictionary for Regulatory Activities) preferred term code for the primary event. Typically a numeric identifier, e.g., '10000060' for 'Gastrointestinal Disorder'."
    },
    "case_completeness": {
      "type": "string",
      "enum": ["Perfectly documented (5/5)", "Well-documented (4/5)", "Moderate documentation (3/5)", "Low documentation quality (2/5)", "Poorly documented (1/5)"]
    },

"report_type": {
      "type": "string",
      "enum": ["Expedited (15-day)", "Periodic", "Annual", "Initial report", "Follow-up report", "None required"]
    },
    "pbrer_inclusion": {
      "type": "string",
      "enum": ["Required", "Not Required", "Optional", "Pending review"]
    },
    "signal_status": {
      "type": "string",
      "enum": ["Under Monitoring", "Not Under Monitoring", "Signal Closed", "New Signal Detected", "Further Investigation Required"]
    },

"expedited_report": {
      "type": "string",
      "enum": ["Submit within 15 days", "Submit within 7 days", "Submit within 30 days", "No expedited report required", "Submission required upon request"]
    },
    "include_in_pbrer_psur": {
      "type": "string",
      "enum": ["Include", "Do not include", "Review for inclusion", "Include if recurring"]
    },
    "update_safety_database": {
      "type": "string",
      "enum": ["Yes", "No", "Pending approval", "Already updated"]
    },
    "update_signal_detection_database": {
      "type": "string",
      "enum": ["Yes", "No", "Pending review", "Already updated"]
    },

"ror_score": {
      "type": "number",
      "description": "Rate of Reporting Odds Ratio (ROR) score for the signal, usually numeric. Example values include '2.5', '1.0', etc."
    },
    "ror_status": {
      "type": "string",
      "enum": ["Above threshold", "Below threshold", "At threshold"]
    },
    "cases_in_database": {
      "type": "integer",
      "description": "Number of similar cases in the database. Examples: 15, 50, 100."
    },
    "labeling_status": {
      "type": "string",
      "enum": ["No update needed", "Update recommended", "Update required", "Pending further data"]
    }
  },
  "required": [
    "status",
    "causality",
    "seriousness",
    "expectedness",
    "primary_event",
    "meddra_pt",
    "case_completeness",
    "report_type",
    "pbrer_inclusion",
    "signal_status",
    "expedited_report",
    "include_in_pbrer_psur",
    "update_safety_database",
    "update_signal_detection_database",
    "ror_score",
    "ror_status",
    "cases_in_database",
    "labeling_status"
  ],
  "additionalProperties": false
}

Sample Screenshot: Analysis

Sample Screenshot: Summary