Home⇛iCONNECT Multidisciplinary Research Journal⇛vol. 3 no. 2 (2026)

Quantifying Semiotic Validity: Measuring Conceptual Depth In Vector Retrieval Systems

Jude Joseph Garcia Jr. | Blaise Howard Bustamante | Bill Ian Alcantara

Discipline: information systems

 

Abstract:

Current vector-based retrieval systems in FAISS, Pinecone, and RAG frameworks rely on metrics (precision, recall, cosine similarity) that measure surface-level matching without assessing whether systems preserve the hierarchical, relational, and contextual structures that constitute genuine semantic understanding. Existing evaluation methods fail to quantify whether embeddings maintain conceptual depth or merely reproduce statistical co-occurrence patterns. This paper presents semiotic validity as a measurable construct, operationalized through the Semantic Depth Index (SDI), integrating four dimensions: geometric coherence (cluster quality), hierarchical structure (taxonomic preservation), relational logic (analogy completion), and interpretive alignment (human judgment correlation). This paper provides a comprehensive mixed-methods design for evaluating seven embedding models, including OpenAI, Sentence-BERT, and BioBERT variants. The study spans dimensions from 384D to 3072D across abstract, technical, and concrete domains, utilizing 500 human-annotated concept pairs validated against established benchmark datasets (SimLex-999, WordSim-353, MEN). The proposed methodology integrates quantitative SDI scoring with qualitative failure mode coding of the lowest-scoring 50 concept pairs to identify systematic patterns of semantic collapse. The framework is designed to compare performance across FAISS (flat/HNSW indices) and Pinecone deployments, testing whether dimensionality, indexing algorithms, and RAG architectures affect conceptual depth preservation. The primary contribution of this work is this framework itself: the first systematic method for diagnosing interpretive fidelity, enabling developers to optimize for semantic coherence rather than similarity alone.



References:

  1. Aly, R., Acharya, S., Ossa, A., Köhn, A., Biemann, C., & Panchenko, A. (2019). Every child should have parents: A taxonomy refinement algorithm based on hyperbolic term embeddings. In Annual Meeting of the Association for Computational Linguistics.
  2. Allen, C., & Hospedales, T. M. (2019). Analogies explained: Towards understanding word embeddings. In International Conference on Machine Learning.
  3. Ammar, A., Koubaa, A., Nacar, O., & Boulila, W. (2024). Optimizing retrieval-augmented generation: Analysis of hyperparameter impact on performance and efficiency. arXiv.
  4. Arseniev-Koehler, A. (2021). Theoretical foundations and limits of word embeddings: What types of meaning can they capture? Sociological Methods & Research, 25.
  5. Artetxe, M., Labaka, G., Lopez-Gazpio, I., & Agirre, E. (2018). Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation. In Conference on Computational Natural Language Learning.
  6. Baroni, M., Dinu, G., & Kruszewski, G. (2014). Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Annual Meeting of the Association for Computational Linguistics.
  7. Bender, E. M., & Koller, A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. In Annual Meeting of the Association for Computational Linguistics.
  8. Berant, J., Chou, A., Frostig, R., & Liang, P. (2013). Semantic parsing on Freebase from question-answer pairs. In Proceedings of EMNLP.
  9. Bolukbasi, T., Chang, K. W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In NeurIPS.
  10. Bordes, A., Usunier, N., Garcia-Durán, A., Weston, J., & Yakhnenko, O. (2013). Translating embeddings for modeling multi-relational data. In NeurIPS.
  11. Brown, K. (2021). The nature of information, semantics, and effectiveness for artificial intelligence and cognition.
  12. Bruni, E., Tran, N. K., & Baroni, M. (2014). Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49, 1-47.
  13. Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (3rd ed.). SAGE Publications.
  14. Eco, U. (1976). A theory of semiotics. Indiana University Press.
  15. Ethayarajh, K. (2019). How contextual are contextualized word representations? In EMNLP.
  16. Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., & Ruppin, E. (2002). Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1), 116-131.
  17. Firth, J. R. (1957). A synopsis of linguistic theory, 1930-1955. In Studies in linguistic analysis (pp. 1-32). Blackwell.
  18. Fu, R., Guo, J., Qin, B., Che, W., Wang, H., & Liu, T. (2015). Learning semantic hierarchies: A continuous vector space approach. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
  19. Harnad, S. (1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3), 335-346.
  20. Hill, F., Reichart, R., & Korhonen, A. (2015). SimLex-999: Evaluating semantic models with genuine similarity estimation. Computational Linguistics, 41(4), 665-695.
  21. Hollenstein, N., Torre, A. D. L., Langer, N., & Zhang, C. (2019). CogniVal: A framework for cognitive word embedding evaluation. In Conference on Computational Natural Language Learning.
  22. Iordan, M. C., Ellis, C. T., Lesnick, M., Lapate, R. C., Cohen, J. D., Norman, K. A., & Baldassano, C. (2019). High-level cortical dynamics may support generalization in visual recognition. bioRxiv.
  23. Lakoff, G., & Johnson, M. (1980). Metaphors we live by. University of Chicago Press.
  24. Le, M., Roller, S., Papaxanthos, L., Kiela, D., & Nickel, M. (2019). Inferring concept hierarchies from text corpora via hyperbolic embeddings. In Annual Meeting of the Association for Computational Linguistics.
  25. Lee, A., Weber, M., Viégas, F., & Wattenberg, M. (2024). Shared global and local geometry of language model embeddings. arXiv.
  26. Lim, J. P., & Lauw, H. W. (2024). Aligning human and computational coherence evaluations. Computational Linguistics.
  27. Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic regularities in continuous space word representations. In North American Chapter of the Association for Computational Linguistics.
  28. Miller, G. A. (1995). WordNet: A lexical database for English. Communications of the ACM, 38(11), 39-41.
  29. Mozolevskyi, D., & Alshikh, W. (2024). Comparative analysis of retrieval systems in the real world. arXiv.
  30. Muennighoff, N., et al. (2022). MTEB: massive text embedding benchmark. arXiv preprint arXiv:2210.07316.
  31. Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016). MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
  32. Peirce, C. S. (1931-1958). Collected papers of Charles Sanders Peirce (Vols. 1-8). Harvard University Press.
  33. Rajaee, S., & Pilehvar, M. T. (2021). How does fine-tuning affect the geometry of embedding space: A case study on isotropy. In Conference on Empirical Methods in Natural Language Processing.
  34. Reif, E., Yuan, A., Wattenberg, M., Viégas, F., Coenen, A., Pearce, A., & Kim, B. (2019). Visualizing and measuring the geometry of BERT. In NeurIPS.
  35. Rosch, E. (1975). Cognitive representations of semantic categories. Journal of Experimental Psychology: General, 104(3), 192-233.
  36. Schnabel, T., Labutov, I., Mimno, D., & Joachims, T. (2015). Evaluation methods for unsupervised word embeddings. In Conference on Empirical Methods in Natural Language Processing.
  37. Senel, L. K., Utlu, I., Yücesoy, V., Koç, A., & Cukur, T. (2017). Semantic structure and interpretability of word embeddings. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
  38. Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., & Farajtabar, M. (forthcoming). The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity.
  39. Speer, R., Chin, J., & Havasi, C. (2017). ConceptNet 5.5: An open multilingual graph of general knowledge. In AAAI Conference on Artificial Intelligence.
  40. Talmor, A., & Berant, J. (2018). The web as a knowledge-base for answering complex questions. In NAACL-HLT.
  41. Thakur, N., Reimers, N., Daxenberger, J., & Gurevych, I. (2021). BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS.
  42. Viswanathan, K., Gardinazzi, Y., Panerai, G., Cazzaniga, A., & Biagetti, M. (2024). The geometry of tokens in internal representations of large language models. arXiv.
  43. Yamagiwa, H., Oyama, M., & Shimodaira, H. (2023). Discovering universal geometry in embeddings with ICA. In Conference on Empirical Methods in Natural Language Processing.
  44. Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., & Liu, Z. (2024). Evaluation of retrieval-augmented generation: A survey. arXiv.

Tools


Copyright © 2026  KITE Digital Educational Solutions |  Exclusively distributed by CE-Logic