An Arabic chatbot folds depression screening into conversation, but the wrong embedder drops retrieval MAP to 4%
Synopsis
Rafiq is an Arabic conversational system for early depression detection that combines per-message symptom monitoring, a three-phase PHQ-9 questionnaire delivered inside the chat, and retrieval-augmented answers grounded in a clinical knowledge base. On 415 held-out instances symptom detection reached 0.98 accuracy, while retrieval quality depended heavily on embedder choice: an Arabic-specific model scored 80% MAP and a general multilingual mixture-of-experts model only 4%.
Interpretation
Arabic conversational depression screening now keeps symptom monitoring, PHQ-9 administration and retrieval-based psychoeducation inside one natural conversation, so users need not leave the chat to fill in a questionnaire. Earlier Arabic mental health applications were largely limited to general information provision or basic chatbot interaction, without structured clinical screening protocols or evidence-grounded response generation. Five licensed psychologists independently rated five conversation scenarios, including suicidal ideation, postpartum depression and adolescent depression, giving a mean overall score of 4.54/5.0 with no critical safety concerns reported.
Symptom detection does not fine-tune a dedicated classifier; a general-purpose model judges each user message against the nine PHQ-9 symptom criteria under keyword-guided prompting. The prompt carries a 217-expression Arabic PHQ-9 lexicon covering masculine and feminine forms and Levantine dialect variants, reviewed by a licensed clinical psychologist. On 415 held-out instances, 365 labelled tweets plus 50 non-depressive controls, accuracy was 0.98 and F1 0.98; keywords appearing in the test set were removed from the prompt dictionary so scores could not come from literal matching.
Responses are generated only after relevant passages are retrieved, so the model answers from authoritative clinical text rather than parameter memory alone. The knowledge base holds 7 Markdown files and 14,624 words curated from DSM-5-TR, MSD Manual, WebTeb and WHO, split along heading hierarchy. RAGAS automated evaluation reported faithfulness 0.88, answer relevancy 0.84 and context precision 0.91; GPT-4o-mini and GPT-4o shared identical retrieval scores, with the smaller model at lower latency and cost.
Perspective
The system targets Arabic speakers, currently mainly Palestinian colloquial Arabic, and is positioned as a supportive companion and screening tool that gives no diagnosis or treatment advice. Clinical evaluation came from five licensed psychologists practising in Gaza, the West Bank and Lebanon, covering five scripted conversation scenarios and 25 ratings, with no patients or real user data involved. The knowledge base covers depression only, not anxiety, bipolar disorder or other common conditions.
Symptom detection was evaluated on Twitter data, which may not represent language in private chatbot conversations, and the clinical evaluation used only five raters, limiting quantitative generalisation. The embedding benchmark is small, with 33 queries and 99 query-document pairs, and its candidate documents were generated by GPT-4 rather than drawn from real clinical text; the mixture-of-experts model was evaluated without its recommended task prefixes, which may partly explain its near-zero scores. Symptom detection does not yet handle negation, which could produce false positives.