I built an open-source AI-powered library for web testing
Posted by p0deje@reddit | Python | View on Reddit | 24 comments
Hey r/Python,
My name is Alex Rodionov and I'm a tech lead and Ruby (and a bit of Python) maintainer of the Selenium project. For the last few months, I’ve been working on Alumnium.
What My Project Does
It's an open-source Python library that automates testing for web applications by leveraging Selenium or Playwright, AI, and natural language commands.
Target Audience
Test automation engineers or anyone writing tests for web applications. It’s an early-stage project, not ready for production use in complex web applications.
Comparison
The closest project I am aware of is LaVague-QA, but it's a test generator (i.e. it generates Selenium+pytest tests from Gherkin specification), while Alumnium is just a library you can use in tests. It uses AI during test execution runtime to figure out Selenium interactions based on what's present in the browser.
Docs: https://alumnium.ai/
Repository: https://github.com/alumnium-hq/alumnium
Discord: https://discord.gg/VDnPg6Ta
mrvalstar@reddit
this is sick!
leogodin217@reddit
My first thought, "I built an AI...." Here we go again. Then, "I'm the lead maintainer for Selenium" Ok, this might actually be good.
It looks fantstic. AI assistance to make testing faster and more interactive. Nice work!
CrankyBlumpkin@reddit
So once you found out he had clout, your perspective changed?
death_in_the_ocean@reddit
Finding out it's not another dumb kid but someone with actual credentials does remove the skepticism, yes.
leogodin217@reddit
100% yes. I see so many AI tools that don't really do much, or don't generalize, or add too much complexity. I don't really expect to see anything interesting. So, when someone with public accomplishments in a critical tool creates something, I am much more interested.
p0deje@reddit (OP)
Being part of Selenium definitely helped me understand the complexity of browser automation. I am not saying Alumnium can do everything as it’s just scratching the surface as of today.
podidoo@reddit
If i understand the examples correctly, you do AI calls to locate elements at every test run.
Why not just using the AI part to generate the tests, so it doesn't rely on the AI bit at each run? It would help we cost, speed and model hallucinations.
p0deje@reddit (OP)
The problem with test generation is that you have to re-generate them every time the UI changes. In my opinion, it's better to do this at runtime, then cache the results and keep using the cached version, only querying LLM again automatically when UI changes.
podidoo@reddit
Yep, that was my idea, having some caching file that you can browse to manually invalidate hallucinations and also that you can share and use directly on your CI.
If you are already doing something around these lines, my bad I missed it (i only checked the repo quickly).
p0deje@reddit (OP)
I have a branch with caching implemented and stored in a separate file, likely will land it in the next version.
juanda2@reddit
Amazing! This reminded me of https://github.com/browser-use/ which was posted a while ago, I tried browser-use a few days ago and it worked very well! However I really like the vision component of Alumnium, very cool! I'm gonna give it a go soon.
p0deje@reddit (OP)
It's quite different from Browser-Use to be honest!
adamnicholas@reddit
Question: can this be adapted for models running locally on something like ollama?
p0deje@reddit (OP)
It definitely can as long as the model would support function calling and structured output. Both are supported in Ollama, I just cannot properly test it since I cannot run Llama 3.2 90B on my laptop.
Toph_is_bad_ass@reddit
Just tried it out and this look awesome.
Quick notes:
I'd rewrite your examples to point towards alumnium.ai instead of Google. I ran a couple of tests while working through your examples and already got hit by the "Are you a Robot?" from Google.
Opt-in or configurable caching (on inference calls) would be absolutely monumental. If it had that we'd absolutely use this at work. I have to imagine that writing tests that cover a large number of pages would be slow & potentially use quite a few tokens.
Does token use increase if the size of the DOM increases? We have some old legacy apps that have absolutely enormous DOMs. These sites would benefit hugely from this (since they're already incredibly difficult to write tests for!).
This is great. I'll be following this project extensively.
p0deje@reddit (OP)
Thank you for your feedback!
Toph_is_bad_ass@reddit
Incredible! I'm excited to show this to my team and get their thoughts. Keep doing great work.
nico_ma@reddit
It compares to https://testzeus.com/hercules doesn’t it? Can you write a summary on why to use alumnium?
p0deje@reddit (OP)
I wasn't aware of TestZeus, but looking through docs it's essentially a testing agent. You feed it Gherkin tests and it does everything. Alumnium is more lightweight as it can be used in existing Selenium/Playwright tests regardless of what test runner you have (pytest, behave, etc.), what test harness you have (e.g. test reporting or parallelization libraries) and without making assumptions about your infrastructure (e.g. it won't start the browser for you).
Oh, and Alumnium works with Selenium too, not just Playwright!
chub79@reddit
I don't quite understand the AI bit. The examples look more like a mini DSL to me than actually something I'd write "in my own language". Cool I guess.
p0deje@reddit (OP)
But this DSL translates instructions into browser actions (clicking, typing, etc.) with the help of AI. You don’t need to locate elements or use Selenium API directly.
chub79@reddit
That's fair. I am curious how come AI was needed to translate that dsl into actions?
p0deje@reddit (OP)
You need something that can take a DOM tree and try to figure out elements and actions to perform on them that are needed to achieve the desired goal. What else except for LLM can you think of that is capable of doing it?
p0deje@reddit (OP)
You need something that can take a DOM tree and try to figure out elements and actions to perform on them that are needed to achieve the desired goal. What else except for LLM can you think of that is capable of doing it?