Where do you find Enterprise gear for <200 users?
Posted by Surge-Monkey@reddit | sysadmin | View on Reddit | 52 comments
Work for a business that seriously needs some enterprise infrastructure. Talking networking and storage servers. (Hint: 200+ devices without vlans)
They’ve made do with consumer switches for so long and there’s anything from infrequent IGMP storms, through to wanting to put yet another 30TB of data onto the “shares” (numerous pcs specifically set up for one function, but are glorified storage). Not to mention they want fast… NVMe storage.
The last 2 times I’ve given them solutions to problems, it required spending money. Like entry level SME money. After that, i ended up with scrounged together 1gbps consumer switches with an uplink dangling from the roof to connect to an adjacent room.
Attempting to try get into the actual Enterprise gear direct from providers is way too expensive. A single switch for $30k vs fitting out the room for $35k
- In Aus- (not opposed to buying overseas, but would need to be local for leasing)
Where do others get their SME equipment from?
I know leasing arrangements is a thing, but haven’t needed to engage with it before because getting the equipment was not really an issue.
I’ve been also trying to find some ex-lease e1.s storage servers as they should be coming off around now.
Trying to find some 100GbE core switches too. We move a lot of bursty traffic, primarily over SMB (yes i know). Current core is 10GbE and 75% of our traffic sources are limited by single cable… yes.
I can fix many problems, but i can’t fix the need to spend money to sort them out and i don’t want to add more pain with more $100 netgear switches.
Last setup i put together was Omada for access and FS for core upgrade (already have FS core)
atnuks@reddit
Hey, I totally get your frustration and bemusement with this as I've been down this road a few times now.
Firstly, I'd say Alta Technologies is worth a serious look even for overseas purchasing. They ship internationally and have been doing refurbished enterprise gear since 1995. This was important for me when starting out as we were setting up a satellite office in Mexico.
Now when it comes to 100GbE core switching specifically, Alta carries refurbished Arista and Cisco Nexus kits for a lot less than if you bought them new. When I dealt with them I found the engineers really know their product lines so you can call them up and say something like "Hey, I need a pair of 100GbE core switches that can handle bursty SMB traffic at this scale" and get a sensible answer from someone who actually knows what they're talking about, not some 23 year old kid spouting sales jargon.
As for the NVMe storage angle, I'd say that the ex-lease market you're already eyeing is the right instinct. Alta buys decommissioned enterprise gear from data centers and large enterprise refreshes... exactly where those e1.s form factor servers tend to surface. The timing you mentioned lines up with typical 3 to 5 year lease cycles so stock should be moving right now.
For VLAN and switching, you're already comfortable with FS for core, which is great because you know what you're getting. I'd say that a refurbished Cisco Catalyst or Nexus for aggregation or core duty is going to run a fraction of the $30k quote you got, and it ships with a warranty and QA, so you don't need to worry too much about failure rates. (This was an issue for me as I had to convince the powers that be that it was OK to buy refurbished gear, something we'd never done before).
If local financing is a requirement you may be able to structure something through Alta or use their pricing as the equipment value basis for a local lease arrangement, but I'd say it's worth asking them directly about that.
The IGMP storm situation with 200+ devices on flat consumer switching is a ticking clock honestly. The good news is that moving to proper managed enterprise gear doesn't have to mean $30k a switch anymore if you're willing to go the refurbished route through a reputable vendor, so I would shoot Alta a message with your spec requirements and see what happens. Worst case scenario you get a free quote and some useful pricing data to take back to management.
Top-Musician4324@reddit
Refurbished is the way to get a decent price (or even an allocation) these days without having the scale of a bigger org. After trying various used dealers, I go with Alta Technologies and ServerMonkey primarily for our servers and switch gear.
screener_kev@reddit
Refurb is your friend at 200 users. Curvature and ServerMonkey are the two we use the most for switches and rack gear, both will quote you on something specific and won't try to upsell you into a 24-port stack you don't need. For storage, Newegg Business and CDW Direct will sometimes match enterprise pricing if you ask. The trick is being honest with them about your seat count, they treat a 200-user shop very differently than they treat a 2000-user shop, and you get better quoting if you say it upfront rather than have them figure it out. For laptops and endpoints, the Dell Outlet Pro Tier 1 deals are sometimes hilariously good if you're patient and willing to refresh the page twice a week.
SevaraB@reddit
100GB will move 2GB per user simultaneously. It sounds like you’re trying to solve inefficient workflow problems (people problems) by throwing more horsepower at them.
Surge-Monkey@reddit (OP)
It’s actually very easy for us to burst that. I can’t go into the specifics, but it’s not uncommon to be moving 100GB files every 15 minutes. Not inefficient, specifically required. But the whole network doesn’t need it, just specific parts.
The inefficient part is the time spent waiting until the transfer can start in the first place.
dustojnikhummer@reddit
I'm gonna guess media production company?
SevaraB@reddit
Or CAD. CAD workstations are notoriously resource-greedy.
ChiefWetBlanket@reddit
Counter to Ubiquiti, Grandstream. The switches are stupid cheap for what you get, their access points are easy peasy to setup, and even their gateway routers do the job effectively. And it can be cloud managed.
For storage, ServerMonkey sells plenty of high capacity servers. Since they don't do anything fancy, get the most disk bays for the cheapest price.
kernelqzor@reddit
Grandstream’s a solid shout for the price, but if they’re already twitchy about “spending real money,” you might want to think about how much hand-holding you’ll need long term. Some of the cheaper stuff saves you up front then taxes your time forever.
For used gear, ex-lease Dell / HPE / Supermicro from refurb shops is usually where the real value is, kind of like what you’re saying with ServerMonkey. In Aus you might have better luck with local refurb places or brokers than trying to import everything, especially if you want any kind of support or leasing.
And yeah, at the scale you’re talking, I’d worry less about “brand is enterprise” and more about “can I get replacement parts fast and does someone else know how to admin this when I’m on leave.”
ChiefWetBlanket@reddit
I wired up a warehouse with Grandstream access points. Because they didn't give us enough lead time to properly do a site survey, we did something crazy. Each row of the warehouse had five APs in the rafters. Since getting back into the rafters was impossible once the racks went up we went with overkill. Every other row had an AP with a directional panel antenna pointing down every row. Those 100 or so APs were still cheaper than a "proper" site survey AP placement. We just turned on the ones that worked best and turned everything else off. With some tweaking we were able to get that place humming. When an AP popped we just turned on the neighboring AP.
Most network gear is set it and forget it. Grandstream is easily as configurable as Ubiquiti just without the cool factor and price to go with it.
Materially_Average@reddit
Ubiquity UniFi
picardo85@reddit
Came here to say the same. This environment is so small that he can just get Ubiquiti off the shelf.
Surge-Monkey@reddit (OP)
The only thing i don’t like about their gear is “some ports this slowed, some that speed, and maybe an uplink” If i want a multi gig switch, give it to me on all ports
dustojnikhummer@reddit
Usually that is a hardware limit, not a firmware/license limit. If a given switch has a combo of 10Gig and 1Gig ports its usually because it's two switches in one chassis
Materially_Average@reddit
High-end Enterprise gear commonly has the same restrictions. Limitation of the chipset used, but that varies per model.
PMURITSPEND@reddit
then pay money.
StanQuizzy@reddit
500+ users across 35 locations all over the US. Unifi / Site Magic working fantasticly!
aCLTeng@reddit
Came here to say the same thing. Running a business about the same size, lots of data flow, enterprise, Fortress Gateway, plus their enterprise campus aggregation switches, you can easily move this much data with a lot less pain and expense.
Witte-666@reddit
This, we are migrating to UniFi in the coming months because of the unsustainable licensing costs. All the people we've spoken to who went from high-end vendors to UniFi were positive so far.
Prestigious-Board-62@reddit
Meraki or Ubiquiti generally depending on how cheap the client is. Meraki is a little more expensive but their support is really good and worth it.
Materially_Average@reddit
I can’t recommend a product (Meraki, or any others) that bricks itself if you can’t afford the support costs.
Yes, you should pay for support to protect yourself from vulnerabilities. And it’s a requirement in a lot of industries. But, some small businesses are barely getting by. They should have their network continue to function if they can’t afford the renewal.
It’s a hostile business model and I refuse to recommend it. It does work well though. And it’s easy to setup. But so is Ubiquiti and you are not tied to their cloud.
dustojnikhummer@reddit
Agreed, buy hardware, not rent hardware.
Surge-Monkey@reddit (OP)
This has been my thought too. EBay and Amazon have been good go-to’s for a bargain.
I only started thinking about the leasing as an alternative to upfront if or were a better incentive for the higher ups.
dustojnikhummer@reddit
I would do what others suggested. Unifi for most of your network and Mikrotik switches for those 100Gbit demands you mentioned
It's a 4 port L2 but only 800 dollars MSRP
https://mikrotik.com/product/crs504_4xq_in
I doubt anyone, incl Unifi, can deliver that sub 1k USD.
IKnowCodeFu@reddit
Ubiquiti should work
rejectionhotlin3@reddit
Mikrotik and ZFS for your storage.
tehiota@reddit
As someone who buys storage by the PB for a geotech company, I’ll bet money your issue is most likely storage IOPS and network flow/design rather than needing 100gbps switching.
That 192Tb array, what encoding method ? How many spindles ? What file cluster size ? Are your reads synchronous or random ? How many NICs in that device ?
You’re probably better off investing in a proper storage platform with lots of spindles, ssd cache , distributed, etc. Isilon was what we used for big data but that was for large files of sequential read/write. The movie industry uses them too. They’re not cheap but they can be found on the secondary market refurbished through the usual players.
Surge-Monkey@reddit (OP)
The 192 is only there as slow archive storage. It’s not designed (and never was) to be fast. Speeds to and from that are fine for its purpose. It’s about 35% allocated for backups, 25% for raw photos and models (3D).
IOPS is a very large factor, which is why traditionally they’ve used consumer NVMe without compression (yes, we live in a situation where a single project -requires- a 2TB drive. Turning compression on tanks the performance and have to wait even longer to start transferring.
Can’t cache part of this traffic, and for the cache we do have, it runs 4x4 NVMe raid0 with 128GB ram backing it. That has 2x10GbE but the only reason that doesn’t cap out is because of so many people being on 1gbps shared cables.
Roughly 12 people : 1x 1gbps cable 12-20x 1gbps cables onto a 1x 10GbE uplink.
It’s not uncommon people to be waiting 8 hours for stuff to move over the network.
tehiota@reddit
Understood. 2TB in my world is still very small for a project. Have you tried running LAG on your uplinks and LACP on. your storage device? That'll give you more overall capacity for more users and could be a quick win.
You haven't said how big your primary storage system is, but when you talk about big data, you talk in systems like Isilon that have multiple nodes with multiple interfaces accessing the same data that's distributed acrossed nodes. You can then distribute the node links to different switches and users get load balanced between nodes. This is good for large sequental reads/writes, not random.
Have you done any actual benchmarking of your IOPS with diskbenchmark or similar to see if you can even max out 1G to 1 User with no other users accessing data ? (after hours)
How you format the drives in clusters also matters, but the answer also depends on the size of the data chunks.
for 12 people, have you considered moving their computers to the datacenter, same switch as the storage device and have them RDP into them from a cheaper device from their desk?
Surge-Monkey@reddit (OP)
There’s only a single piece of cat6e that connects some of the heavier traffic users. LACP requires more cables to be run, which is when i said run fibre, not Cat.
LACP configured on what i can. Between switches that are capable of running out and servers that can use it (cache).
The “primary” storage is too distributed to really know. If it were centralised onto an all flash NAS, I’d estimate about 30TB once i make use of snapshots to provide instances of data. Otherwise that single 2TB disk is multiplied by 5. Each project has 5 copies of data. Then about 9 projects.
So duplicated data, a lot. Stored properly, a fraction.
As far as benchmarking, can absolutely handle over 1gbps. We frequently max out the 10GbE link that goes to each of these “servers” and to staff that also have 10GbE NICs.
But it does mean that one person can slow other people down. Just not enough bandwidth. That is actually the only saving grace of having this storage over multiple machines… they don’t tend to grab data from the same machine at the same time. So we get more throughput that way.
Everything is local, no DC. Think of it more as a glorified home lab running in production…
tehiota@reddit
LACP won't typically help at the user level. it's only good for different source/destination MAC addresses. I was saying to use it on your switch uplinks.
Call my skeptical, but I just don't see how you can max out these 10G links to desktops from a non-optimized setup. Without detailed information, and a diagram, i'm not sure there's more help I can offer. I don't think throwing more network is going to solve your issues though.
Surge-Monkey@reddit (OP)
The switches that are capable of doing LACP have it enabled. Luckily the “core” switches are running 40gbps between themselves in the racks. But can’t do LACP anywhere else because of dumb switches and not enough cables (which would require running through spaces that require an electrician)
When you have 40 people sharing a single 10G link, 80 people sharing a 10G link and another 60 odd who’s are actually using about 12x10G links.
Separate to them are the actual machines with this “storage” which are primarily for a CI/CD pipeline that produces these 100GB packages. If we offload that storage from local to NAS to save on storage costs, it all starts to add up bandwidth wise, especially accounting for using NVMe-oF for the CI/CD storage
benuntu@reddit
Unifi for switches, gateways, APs. Definitely not top-tier, but also doesn't have a top-tier price and requires no recurring contracts or fees. 10gbe should be fine if you have it set up correctly and not bottlenecked at a storage server. Not sure what your budget is, but obtaining a 2U server with 12x 3.5" drive bays, 4x10gbe networking and installing TrueNAS would be a good upgrade in speeds. You can add NVMe drives if you want, but it's more cost effective to add more RAM(ARC). The only caveat to that being if you have a lot of repetitive reads, and then you'll want to add a faster secondary cache(L2ARC).
I'm running several of the Unifi Pro XG 48 PoE switches for general connectivity, which provides 2.5gbe to all ports. 32x ports have 10gbe with 4xSFP28 ports delivering 25Gbe. Those 4 ports are perfect for running back to your core switch or connecting to your storage servers. Pair these switches with a Pro XG Aggregation switch that has 32x 25gbe SFP28 ports. Or go one step above if needed with the Enterprise Campus Aggregation switch.
bunnythistle@reddit
If you are seriously, strongly constrained on pricing and every penny counts, then Mikrotik may be worth looking at for networking - their prices tend to be very low compared to other options, but their products are still pretty high quality and feature rich.
However, also compared to other options, Mikrotik's products come with a much steeper learning curve.
Surge-Monkey@reddit (OP)
Have a few mikrotik already and have Winbox on my taskbar :) i saw the 100GbE switch they released in the last 12 months. It’s great, just not enough ports. :(
Jeff-J777@reddit
I would look into Ubiquiti. Or Meraki or Aruba. I don't know why you would spend 30k on a single switch. I have done large 800+ user companies with 100+TB of data. Ran the core network on 10G fiber links and everything was fine we never taxed the connections.
I don't know why you think you need 100GbE core switches, seems like way overkill to me.
Surge-Monkey@reddit (OP)
2 reasons. Every minute waiting for 100GB file to transfer means people are doing not a whole lot. When you have 30 people trying to copy those files at the same time… The other reason is to make use of NVMe-if to phase out having to have individual drives in each of these other machines. Still needs to have Gen3 pie speeds, just without the local drives. (Maybe one drive as a local cache)
nVME_manUY@reddit
Ubiquiti and/or Mikrotik
Surge-Monkey@reddit (OP)
We’ve got some mikrotik already, they’re definitely good. I just wish their offerings had more ports.
BonezOz@reddit
First, where in Australia?
From the sounds of it your SME is cheap, short hands, long pocket problems.
As someone said, Ubiquiti for network gear, cheap(er) cheerful enterprise class switches, routers, and APs. I'd still recommend a high end firewall for anything coming into the network from outside.
Setup a partnership with a wholesale distributer like Dicker Data. This way you can get components at wholesale and not retail prices.
Replace your storage PCs with a good sized NAS, stick with enterprise class SSDs for primary storage, but keep a few SAS drives set aside for long term storage, and rent a space in a datacentre somewhere, even if it's just enough room for a firewall, switch, and NAS so you can send backups offsite.
And you need to tell the higher ups that if they want data security, they're going to have to spend money, there's no way around that. Maybe call a MSP that specialises in SMEs to help. I can point you to a few good ones depending on your location 😉
Surge-Monkey@reddit (OP)
I’ll say southern Australia.
Trust me, i want the NAS. We can save a very significant amount of storage moving to this and dedup will immediately happen as a result. It will solve a lot of headaches.
Yeah I’ve been looking at the U.2 and E1.s becoming available now. I did look at Dicker Data, but wasn’t sure if we’d buy in the volume or if ours worth the hassle going through the process only to find out that i can’t get approval to buy what’s needed anyway 😑
Firewall is covered at least. Backups have 2 off sites. And unfortunately, normal working data can’t live outside the local network :( even with a 1gbps symmetric link, that wouldn’t be fast enough 😅
dustojnikhummer@reddit
Unifi + Mikrotik. Mikrotik has some quite cheap 100Gbit L2 switches, but I hope they don't want to route at that speed.
Accurate-Ad6361@reddit
Your Problem is not uncommon, some suggestions:
If your needs are Met with fileshares and Active Directory, go refurbished: San Disk and Samsung read intensive SAS drives go for 100-150€ per 1.92 TB pushing up to a gigabit per second. Pair that with R730 servers and some ram on Proxmox / TrueNas and you can go crazy using Mellanox Connect-X and Mellanox SN-2010 switches (I caught two for 3000 total).
Dell HBA330 + Server should go for 300-400 USD plus Ram.
Some notes: - check that shroud, faceplate and key are present.
chippinganimal@reddit
I work for a small nonprofit broadcast TV station with similar storage and networking needs, ubiquiti and mikrotik are great for cheap and actually fast switching, and refurbished Dell servers such as the r740xd2 can be had cheap and equipped to do NVME on its backplane.
I believe you can find supermicro 847 4u 36 bay cases for pretty cheap refurbished as well if you want to stick with ATX standard hardware.
Just be aware how expensive ddr5 rdimms have gotten recently if you decide to go with newer epyc or xeon based systems
khobbits@reddit
You might be able to get away with Mikrotik switches? Those are pretty cheap, have both a webgui and terminal. Offer a subset of enterprise features.
For storage, it depends on your comfort levels, really. You will probably be able to find some cheap supermicro chassis with nvme, but you'll need to set up the storage yourself. Either install zfs or ceph, or use a 'nas distribution' like truenas scale.
If you want more hand holding, I guess things like qnap?
_AngryBadger_@reddit
I have a lot of TP-Link Omada stuff at various clients and it works really well. I have a self hosted controller that makes life really easy.
fedesoundsystem@reddit
Get some old server than you can shove a lot of cheap disks, install Windows (truenas requires lots of ram) and make some storage pools with some redundancy. Then wait a while and repeat, and make a dfs server
Surge-Monkey@reddit (OP)
Have a number of these already. But drives without dram are too slow. We write over a few hundred GB at a time and blow out the plc cave on ssds. I have a 192TB archive array on truenas with 7.2k SAS disks. That’s not the issue though. Were a little beyond that point now
oliland1@reddit
We had a customer with similar requirements.
We gave a try to the ECS-Aggregation from Unifi and its been working very well and it’s very inexpensive.
czj420@reddit
Netgear has better switches that didn't cost as much as some
sexaddic@reddit
Ubiquity is the only realistic option here.
RedShift9@reddit
I buy second hand Cisco equipment. Company doesn't want to spend the dough? No problem if you can live with some scuffs on your switches and servers.