The allure of generative AI's apparent competence has led many researchers to venture into unfamiliar territories, applying these tools to domains where they lack the necessary expertise to critically evaluate the outputs. This phenomenon represents a fundamental departure from traditional research practices, where domain knowledge serves as the cornerstone of methodological rigour and result interpretation. The consequences of this shift extend beyond individual research projects, potentially undermining the broader scientific enterprise through the propagation of unvalidated or erroneous findings. Central to addressing these challenges is the recognition that domain expertise is not merely beneficial but essential for the reliable evaluation of generative AI outputs (Asamoah et al. 2024). The complexity of modern AI systems, combined with their propensity for producing plausible yet potentially inaccurate results, necessitates a fundamental reconsideration of how we approach AI-assisted research. This analysis demonstrates that domain expertise is indispensable for validating generative AI outputs, requiring researchers to exercise heightened caution when applying AI tools outside their areas of competence. The imperative extends to implementing robust validation protocols that prioritise human oversight and critical evaluation of both AI processes and outputs, ensuring that the promise of AI-assisted research does not compromise the fundamental principles of scholarly rigour.
The role of domain expertise in evaluating generative AI outputs transcends simple fact-checking to encompass nuanced understanding of contextual appropriateness, methodological soundness, and disciplinary conventions (Dash et al. 2022). Gallegos et al. (2024) provide compelling evidence for the multifaceted nature of bias in large language models, demonstrating that effective evaluation requires not only technical understanding of AI systems but also deep knowledge of the specific domains in which these systems operate. Their comprehensive survey reveals that bias evaluation must account for different levels of model operation—embeddings, probabilities, and generated text—each requiring distinct forms of domain-specific expertise to assess effectively. The interpretability challenge represents a particularly complex aspect of domain-expert evaluation (Bayer et al. 2022). Unlike traditional research tools whose limitations and biases are well-understood within specific disciplines, generative AI systems operate through complex neural architectures that obscure their decision-making processes. Domain experts must therefore develop new competencies that combine their existing disciplinary knowledge with an understanding of AI system behaviour. This requirement extends beyond surface-level output evaluation to encompass critical examination of the underlying processes that generate AI responses, including training data provenance, model architecture influences, and potential sources of systematic error.
The inadequacy of general AI evaluation metrics becomes apparent when considering domain-specific requirements for accuracy and appropriateness. Standard metrics such as perplexity, BLEU scores, or coherence measures may indicate technical proficiency whilst failing to capture domain-specific errors that could fundamentally compromise the validity of AI-generated content (Chang et al. 2024). For instance, a generative AI system might produce a scientifically coherent-sounding explanation of a biological process that contains subtle but critical errors in mechanism description, errors that would be immediately apparent to a domain expert but might escape detection through general evaluation protocols. Furthermore, the dynamic nature of knowledge within specific domains necessitates ongoing expert involvement in AI evaluation processes. Scientific understanding evolves continuously, with new discoveries potentially invalidating previously accepted theories or methodologies. Domain experts possess the contextual awareness necessary to identify when AI-generated content reflects outdated or superseded knowledge, a capability that cannot be replicated through automated evaluation systems. This temporal dimension of expertise underscores the irreplaceable role of human domain knowledge in maintaining the currency and accuracy of AI-assisted research outputs.
Researchers using generative AI tools must avoid venturing into domains where they lack sufficient expertise (Peskoff and Stewart 2023). When working with AI outputs, particularly in analytical contexts, researchers must examine the underlying processes by opening analysis sections and reviewing the code to understand what the model has done. If researchers lack the domain expertise necessary for validation, they must not blindly trust the model's outputs, regardless of how confident these appear. AI models can make errors, producing plausible-sounding but fundamentally incorrect results that only knowledgeable human oversight can identify. Effective AI oversight requires active validation rather than passive acceptance of outputs. This involves systematic examination of AI results against domain-specific criteria, verification of methodologies, and critical assessment of accuracy. Researchers must engage with AI systems as tools that augment rather than replace human expertise, maintaining essential human supervision for checking and validating all AI-generated content. Without proper validation protocols and domain expertise, the apparent sophistication of AI outputs can mask underlying inaccuracies or methodological flaws (Peskoff and Stewart 2023).
In sum, domain expertise is indispensable for reliable evaluation of generative AI outputs, and researchers must resist the temptation to apply AI systems to unfamiliar domains without adequate validation capabilities. Generative AI systems are sophisticated tools that require expert guidance rather than autonomous agents, necessitating systematic validation protocols and human oversight to identify errors, biases, and limitations that automated systems cannot detect. The research community must implement robust validation mechanisms and prioritise domain expertise in AI evaluation processes to harness AI capabilities whilst safeguarding research integrity and scholarly knowledge.
References:
1. Asamoah, Pasty, Daniel Zokpe, Richard Boateng, et al. 2024. Domain knowledge, ethical acumen, and query capabilities (DEQ): a framework for generative AI use in education and knowledge work. Cogent Education 11(1): 2439651. Available at: https://doi.org/10.1080/2331186X.2024.2439651 ^ Back
2. Bayer, Sarah, Henner Gimpel, and Moritz Markgraf. 2022. The role of domain expertise in trusting and following explainable AI decision support systems. Journal of Decision Systems 32(1): 110–138. ^ Back
3. Chang, Yupeng, Xu Wang, Jindong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15(3): 1–45. ^ Back
4. Dash, Tirtharaj, Sharad Chitlangia, Aditya Ahuja, and Ashwin Srinivasan. 2022. A review of some techniques for inclusion of domain-knowledge into deep neural networks. Scientific Reports 12(1): 1040. Available at: https://www.nature.com/articles/s41598-022-05085-9 ^ Back
5. Gallegos, Isabel O., Ryan A. Rossi, Joe Barrow, et al. 2024. Bias and fairness in large language models: A survey. Computational Linguistics 50(3): 1097–1179. ^ Back
6. Peskoff, Denis, and Brandon M. Stewart. 2023. Credible without credit: Domain experts assess generative language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 427–438. Available at: https://aclanthology.org/2023.acl-short.47/ ^ Back
