Small Models, Precisely Tuned: A Two-Tier Small-Language-Model Architecture for Autonomous Penetration Testing Soft-Prompt-Tuned Strategic Reasoning Paired with a LoRA-Fine-Tuned Tool-Calling Executor, Evaluated Against VAmPI and OWASP Juice Shop

Vidhaan Appaji; Lokesh Upputri; Senthil Arasu T; Daksha S

doi:10.69968/ijisem.2026v5i4229-243

Authors

Vidhaan Appaji Master of Science in Applied Artificial Intelligence, Shiley-Marcos School of Engineering, University of San Diego
Lokesh Upputri Master of Science in Applied Artificial Intelligence, Shiley-Marcos School of Engineering, University of San Diego
Senthil Arasu T Master of Science in Applied Artificial Intelligence, Shiley-Marcos School of Engineering, University of San Diego
Daksha S Master of Science in Applied Artificial Intelligence, Shiley-Marcos School of Engineering, University of San Diego

DOI:

https://doi.org/10.69968/ijisem.2026v5i4229-243

Keywords:

Agentic Penetration Testing, Large Language Model Fine-Tuning, LoRA-Based Vulnerability Exploitation, Autonomous Security Testing, Multi-Agent AI Architecture

Abstract

Large language models increasingly serve as reasoning engines inside automated penetration-testing agents, but two problems recur across the recent literature: general-purpose models used as end-to-end planners struggle to sustain coherent multi-step attack reasoning, and giving that reasoning role to a single large model is expensive per call. This project addresses both by splitting the pipeline into two small, purpose-trained language models instead of one large general-purpose one. A Strategist Qwen3-8B adapted with soft-prompt tuning on 2,311 hypothesis-generation examples spanning 26 real vulnerability categories mined from disclosed HackerOne reports - proposes which vulnerability to test next and why. An Executor Qwen3-1.7B-Base adapted with LoRA - turns that hypothesis into a concrete, correctly formatted HTTP tool call. Both models were evaluated in isolation and inside a closed, autonomous loop against two live, intentionally vulnerable targets: OWASP Juice Shop and a self-hosted VAmPI instance. The headline result is a controlled ablation: holding the Strategist and the attack loop fixed and swapping only the Executor's weights, LoRA fine-tuning takes the autonomous loop from 0 of 9 target vulnerability categories confirmed to 7 of 9. Combining that loop with a separate three-role harness run resolves 8 of VAmPI's 9 applicable categories overall. Across all 38 applicable tests pooled from both live targets, 22% produced a confirmed, evidence-backed finding, while the soft-prompt Strategist matched or beat a fully-prompted master-prompt baseline on the same base model on 11 of 18 comparable vulnerability patterns while using 59–85% fewer input tokens per call. The project is equally direct about where the approach struggles: against a frontier model with no scaffolding the Strategist scores 2.53/5 versus 4.94/5, and multi- step reasoning chains degrade to a mean score of 2.0/5 by the fourth or fifth step. The results support a narrow but load-bearing claim: fine-tuning the execution layer of a small-model agentic pipeline is, in this setting, a larger and cheaper lever on end-to-end task success than fine-tuning - or scaling to - a larger reasoning model.

References

[1] American Psychological Association. (2025). In-text citations. https://apastyle.apa.org/style-grammar-guidelines/citations

[2] Andreucci, F., & Castiglione, A. (2026). Automation-Exploit: A multi-agent framework for risk-mitigated automated exploitation using digital twins. arXiv preprint.

[3] Checkmate authors. (2025). Checkmate: A Planner-Executor-Perceptor architecture for automated penetration testing. arXiv:2512.11143.

[4] Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., & Rass, S. (2024). PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In Proceedings of the 33rd USENIX Security Symposium.

[5] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-rank adaptation of large language models. arXiv:2106.09685.

[6] Lester, B., Al-Rfou, R., & Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 3045–3059).

[7] Moonshot AI. (2026). Kimi K2 technical report and model documentation.

[8] PentestGPT v2 authors. (2026). Task difficulty assessment and evidence-guided attack tree search for automated penetration testing. arXiv:2602.17622. reddelexc. (2026). hackerone-reports: A curated dataset of publicly disclosed HackerOne vulnerability reports [GitHub repository].

[9] Shen, X., Wang, Y., Chen, Z., & Liu, J. (2025). PentestAgent: Incorporating LLM agents into automated penetration testing. In Proceedings of the ACM Asia Conference on Computer and Communications Security (AsiaCCS).

[10] Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., & Hashimoto, T. B. (2023). Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models.

[11] Xu, L., Xie, H., Qin, S. Z. J., Tao, X., & Wang, F. L. (2023). Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint.

[12] Yang, J., Prabhakar, A., Narasimhan, K., & Yao, S. (2023). InterCode: Standardizing and benchmarking interactive coding with execution feedback. In Advances in Neural Information Processing Systems, 36.

[13] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR).

[14] Zhang, A., Perry, N., Dutta, R., Kang, D., & Zou, J. (2024). Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. arXiv preprint.

[15] Zhu, M., Wang, Z., Li, K., & Anastasopoulos, A. (2024). NYU CTF Bench: A scalable open-source benchmark dataset for evaluating LLMs in offensive security. In Advances in Neural Information Processing Systems, 37.