How do you manage a software evaluation?
Posted by Wolpertiing@reddit | sysadmin | View on Reddit | 35 comments
Just finished a 4-month eval of 5 platforms. Coordinated demos, tracked quotes across several rounds of negotiation, logged email threads with 8 different reps, and tried to build a coherent deck for leadership at the end.
Ran all of it out of a spreadsheet and Gmail labels.
Curious how others handle this. Is there a tool people are actually using for the buyer side of this? Not G2 for finding software, I mean for managing the eval once you have a shortlist. Contacts, notes, quotes, demo summaries, etc
Or is everyone just using Excel?
OneSeaworthiness7768@reddit
I’m gonna go out on a limb and say you’re working on one right now.
hkusp45css@reddit
I do 3 vendor selections for every platform, I have swapped or stood up 19 platforms in the last 24 months. None of what you're describing is part of my processes.
We interview the platform offerings, we select the vendor who has the best platform for our needs, and then we negotiate pricing.
By the time we're onboarding the vendor, I've got about 4 hours of work in every platform. 3 demos and a requirements and constraints meeting with the selected vendor.
It's just not that big of an issue, in every org I've done this for.
phoenix823@reddit
This is much closer to my approach as well. Perfection is the enemy of the good, especially when I'm only looking at a 1 year contract.
1) Make sure it will accomplish all my major goals
2) Pay a reasonable price for it
3) Implement and get the business value
Darkhexical@reddit
Point 1 is the key. If it doesn't accomplish your goals go to the next vendor. Don't get hung up on the sales pitch if it doesn't do what you actually need it to do.
Wolpertiing@reddit (OP)
this is basically what i do, but i always have to consolidate the data somewhere so i have it all in one place or i will 100% lose things - only thing ive found for that is excel. is that not an issue for you or do you track the process a different way?
Wolpertiing@reddit (OP)
That sounds wickedly efficient, how long did it take you to develop this process? when you first started evals for product selection, was it this streamlined?
hkusp45css@reddit
I read the requirements for best practices related to VDD and onboarding and implemented them.
It wasn't like I invented fire.
Wolpertiing@reddit (OP)
i use those requirements as well but i still have to track the responses from venders on the questionnaires, the checklists as well as my own due diligence checks on each vendor i look at. you may not be inventing fire but if you're saying you stood up 19 systems, and looked at 3 systems to decide on each of these 19 and you also implemented them...all in 24 months - you're either doing this full time or a robot.
hkusp45css@reddit
It sounds like you've decided on a bunch of forms and processes that are making the work suck.
Try to simplify the process so it only contains what the process needs to be successful.
Wolpertiing@reddit (OP)
i just use excel and my inbox - i think my problem is all the forms i recieve back from the vendors lol...ill take your advice and try to keep it simple, thanks for your input
man__i__love__frogs@reddit
I work as an EA now, and previously did engineering/solutions work but its a smaller org so I still wear a few hats. Also forgive my ADHD here but much of this applies to both how you should set up a workflow/system to evaluate tools, and how you should evaluate tools going forward. They're kind of one in the same.
No shame, I use(d) LLMs to get started on my evaluations. They're great at getting started on feature and strategy alignment.
We have architectural principals tied into our digital intake process. Basic things like due to our size/industry and ever growing overhead of supporting custom solutions, SAAS -> PAAS (AVD+AzureSQL), on-prem out of the question. Consider source of truth for data. Is this tool going to have or require data from another system, how will they avoid drifting apart? Who is the business owner? What is your current maturity in this process? Balance using what you already have (sharepoint lists, power automate, etc...) versus the complexity of custom builds versus SAAS.
Most importantly, goals based initiatives. You need to understand what exactly what you're looking for in a tool that will help you with this process (importantly like I said earlier, this is not just for the this process in setting up a way to evaluate, but also how you should evaluate things going forward), and how its going to accomplish that. Not just go looking for tools that can do xyz and decide which one you 'like' best. That is backwards tool-first thinking and how you end up with some crazy complex enterprise tool that has an insane barrier and rigidity to configuration and you spend all your time just learning how to adapt your processes to the tool. Or you find out that it can't do a bunch of things you want it to as you learn your process. Or how you end up with something that no one really understands or wants to use and it sits on a shelf. You cant buy maturity in a business or IT process just by going out and purchasing a tool.
So since you're already familiar with how the Outlook + Excel thing is working write out what your pain points and struggles are. Make some measurable goals/outcomes to improve that and then start evaluating tools around that. Get a LLM to spit out a matrix of all these things and score the tools.
Wolpertiing@reddit (OP)
I have a solid grasp and understand of what I need in a tool and probably do too much cross comparison and a matrix check to make sure I'm not being biased. But the research always comes after I've identified pain that tech can solve and ruled out using existing infrastructure. For me the research is one part of the bigger project that is 'solve for x pain'
I just wish I wasn't doing it in excel
man__i__love__frogs@reddit
That's totally fair, just you had mentioned in another reply that you're worried about picking the wrong tool and how that ends up affecting things...
In my experience tools end up being the wrong one for those reasons mentioned above. I guess I would need to know if anything had failed, why did it happen? Or that hasn't happened yet, you're just worried about it, imposter syndrome? If it's the latter, that's what the right processed, supported by existing frameworks helps with, and identifying pain points, inefficiencies and other considerations with stakeholders and teams that will be using it. With the right approach you sort of take the subjectivity out of it. If something doesn't work, then things were identified wrong, or something is wrong with the process, and you make a plan to improve based on what went wrong.
Like I said the goal can't be just to get the tool operational, you need measurable outcomes with business value. When the tool is implemented you then need to go back and do a post mortem, was the initiative successfull in bringing in that value it was supposed to or not and then work on identifying and improving things from there.
SikhGamer@reddit
I make the first gate to be "the things we NEED" that cuts out 90% of the crap vendors.
SuperfluousJuggler@reddit
We are a google shop so we just make folders and put relevant information into them. Once its decision time I toss it all in NotebookLM and start querying against it and put them against each other with our unique findings until we find a winner.
Or if we did have a winner in mind, we toss the whole pot into NLM make any notes on them, convert to sources and link all that to a GEM and then push in our compliance, risk, scope, etc and see if it would really work or a competitor would be better. Always good to remove the rose glasses for a true evaluation, Gemini is cold and ruthless when it comes to that, well we kind of made her that way.
This shaves weeks off the process and all we need to do is verify the output is correct by doing quick lookups with the source material.
Wolpertiing@reddit (OP)
This is super helpful - I can see the advantages of using NLM to short cut through the vendor data - gonna have to do this next time, thank you!
Sharp_Animal_2708@reddit
the tool matters less than defining decision criteria and weightings upfront. ive seen evals drag for months because every demo just added more questions instead of narrowing. did you have a scoring framework or was it more vibes based?
Wolpertiing@reddit (OP)
I have a scoring matrix - must have capabilities, budget, ability to scale, overall ROI, flexibility (what else could it solve if anything), review score (i used g2 mostly for this) I used an llm to help me decide the weights of each and I feel like i may be forgetting a couple in that list but I get what you're saying. I'm always concerned I'll be biased to the prettiest thing that turns up - I'm a sucker for a good UI.
Frothyleet@reddit
I imagine it boils down to what business needs you actually have to solve here (e.g. what your procurement bureaucracy looks like, what documentation your leadership will want, who all is involved in the decisionmaking, and so on).
Wolpertiing@reddit (OP)
I present to the director of the department who then brings it to the executive team for final write off - so my part is more of a, 'what did you look at, what made you choose this, what is the cost, what is the ROI we expect, what it the amp timeframe, how is data migration handled, it is compliant." i present in a slide deck during a zoom meeting usually.
my directors part is more of a "approve this in my budget please, yes we are sure we need this"
i think the reason why i may be overthinking the research is because, if i pick the wrong tool, it affects me, my team and my director - so making sure i pick right gives a bit of anxiety
man__i__love__frogs@reddit
How you do that, or rather the guardrails that are put up in the process to lead you to the right solution is the field of enterprise architecture.
There exists a whole framework called TOGAF that helps with that. TOGAF is excessive, but it has great methodology for software intake processes.
Frothyleet@reddit
Well, with any tool adoption, you're gonna find out things after the fact. And not always pleasant things.
But it sounds like you are going above and beyond to explain your reasoning behind your recommendation, which means you can always point to that in the future.
Wolpertiing@reddit (OP)
I am nothing, if not thorough - thanks for your input : )
ViolinistBusy9070@reddit
Wolpertiing@reddit (OP)
i'll look at airtable - im right tired of trying to find workarounds but if its better than labels im for it, thanks!
ViolinistBusy9070@reddit
Airtable's linked records are the key once vendors, contacts and quotes are connected it stops feeling like a spreadsheet and actually becomes a system. Worth the setup time once you're doing evals regularly.
dennisthetennis404@reddit
Notion works well for this, one database per vendor with linked pages for contacts, demo notes, and quotes, plus a master view to compare them side by side when it's time to build the leadership deck.
Wolpertiing@reddit (OP)
Master view you say? Didn't realize Notion had this
EndpointWrangler@reddit
Mostly everyone is using Excel or Notion, honestly. For something more structured, Airtable works well for tracking vendors, contacts, quotes, and status in one place with linked records, it's close enough to a spreadsheet that there's no real learning curve but gives you relational views that Gmail labels can't.
Wolpertiing@reddit (OP)
Ive got a colleague that uses Notion every day but it didnt give me the full picture i was looking for - airtable seems to be the answer as long as you put in the assembly time
cbtboss@reddit
Yes to excel for what we use for tracking/evaluating vendors.
Wolpertiing@reddit (OP)
excel is carrying enterprise IT on it's back - thanks for the input!
Hennaj69@reddit
You’re overthinking it.
Wolpertiing@reddit (OP)
I mean...you're not wrong..
BrainWaveCC@reddit
We're using Excel.