Solo IT in a medium size factory (300 employees)
Posted by shimonj1@reddit | sysadmin | View on Reddit | 58 comments
So, here's my story:
I'm a computer technician with just a few years of experience. Recently I was fired from my job at the hospital and started working as an IT/technological support person in a factory with 300 employees.
This means I'm responsible for managing everything, including sysAdmin stuff which is new to me. I can get a one on one mentoring from outsourced guy who has access to all servers and can explain a bit but I figure i need to run everything by myself, since I have all the time in the world to learn the system and figure things out.
The atmosphere here is pretty chill. for now atleast LOL.
How do you suggest I approach this?
Ethan-Reno@reddit
Verify backups, learn your stack, see what you’re responsible for, ideally…
MidnightBlue5002@reddit
this. period.
Frothyleet@reddit
Scared I had to scroll down for this! Absolute number one priority for any new environment should be backups - where are they, are they working properly, can they be restored properly.
Elensea@reddit
What’s expected from you? Crazy how these companies constantly hire an IT manger under the guise of IT technician.
Ok-Double-7982@reddit
Chill atmosphere in an environment with 300 employees? How many computers are there, like 10 kiosks? I'm baffled unless everything is so old and out of date like running on Windows 10 or XP that everything seems chill because nothing ever changes lol.
-King-K-Rool-@reddit
I mean a 300 employee factory is pretty small. Key part here being that its a factory. You very well may not be too far off, probably 10 kiosks running proprietary OS that he doesnt have to deal with, a few HR computers, a few exec computers, and maybe some laptops for department managers. Hes probably got less than 50 actual end user pc's total to manage.
Arudinne@reddit
Yeah, a 300 employee factory is very different from a 300 employee office.
Bluetooth_Sandwich@reddit
*unmanaged switch shoved in a ceiling tile goes burrr*
Ok-Measurement-1575@reddit
Chill untill all the SCADA nonsense that was sellotaped to the ethernet network stops working :D
Cecil4029@reddit
Chill until the random profibus cable connecting PLC's craps out and the line goes down lol
Ok-Measurement-1575@reddit
Shielded cables? Ain't nobody got time fo dat. Slap these utp cables in!
dreniarb@reddit
For me that's when I contact the outside scada vendor. For me when it comes to scada my responsibility stops at the network jack.
shimonj1@reddit (OP)
I don't know yet really. I am here for a week doing nothing but unlcoking users or creating them for new employees.
Joy2b@reddit
It’s about time to do some network mapping and inventories.
Give your outsourced person a heads up before you scan a network. If I am in a rush, 1 hour, if not, maybe 2 business days. They should see the email before they see alerts.
First you identify the physical hardware, then the apps, then the backups.
Notice the versions, what needs updates, what are the dependencies.
Pair your hardware list with pictures of how to find that hardware, and where you keep any spares. Your most important wireless access point may be tracked down by following the Ethernet cable going up to a panel in the ceiling.
Look for physical risks. If there’s water marks on that panel, you want to cultivate a good relationship with maintenance, make sure they’re comfortable calling you if there’s a water problem so you can relocate equipment.
As an FYI, factory machines are often on specialty hardware that can tolerate dust and vibration better, and doesn’t get replaced without real budget planning. You may still need a low static air cleaning routine.
Some machines in a factory may need a bit of extra network security. If a machine worth 10 times your house is running Windows 98 and can’t be upgraded, you care about the firewall and subnets and you’re probably curious about the network alerts.
PerfectAverage@reddit
It sounds like you should be more proactive, and understand the environment you own.
That way you can plan for future success.
jaydizzleforshizzle@reddit
Right? I’ve got friends in IT and I’ve stopped prodding, but it’s obviously a huge gap. I’ll ask my noc friend about anything and he doesnt have a clue about his environment, its like “i get you aren’t an architect but dont you wanna know how it works?”, but thats the huge gap i see in a lot of IT people, they either wanna chill and collect their paycheck or they are curious and that makes them better at IT.
Remarkable_Spell6058@reddit
His previous job was at an hospital, reading some comments here on that environment suggests that ANYTHING is "chill" compared.
Terriblyboard@reddit
How many PCs are actually deployed in the company? Is there an Operational Technology person that deals with the machines or is that also you? I would first inventory and document everything. Start documenting the IT processes as well. Make sure you have good backups and failback plans in place.
Flabbergasted98@reddit
"Hey google, what is a healthy ratio of IT to staff?"
bourbonandpistons@reddit
Document and eslcalate issues in writing to your superior.
"No backups on x,y,z. I need a,b,c to make it happen"
Do it for compliance. Redundancy for mission critical things. Etc..
When something goes wrong
"I addressed it with you on date/time"
SpaceChimps98@reddit
I would also document vendors and ISPs and anyone else involved with support so you know who to contact when something breaks. If you're the only IT guy, you're going to be getting calls for anything electronic that is broken and if you don't fix it, you'll at least want to know how to document and hand it off to whomever does.
-King-K-Rool-@reddit
If you can swing the spare time for it, start making the entire network your own, documenting everything and learning along the way. Pick out your preferred documentation app and start fresh documentation of everything from the ground up. This can actually be a really fun long term project that will build you a ton of skills over the next 5 years.
Map out your entire network so you know what you're working with, take inventory of every device, track down every pc, ap, server equipment, network equipment, what cabling youre running and where, map out and list everything. Getting a decent inventory management software will help this a lot too.
Start making everything your own, rename everything names that make sense to you, build your own conventions, replace things you dont like with things you do like. Set up your own SOC, your own endpoint monitoring, your own repair flow.
You might be able to talk whoever is in charge of budgeting into nabbing you a Udemy Pro account or something similar that you can use to learn about topics youre not strong in and just teach yourself as you go.
SpaceChimps98@reddit
And create a ticketing system and force people to use it. Nothing worse than trying to configure a switch or run cabling and then some jackass decides to start calling randomly because the "r" key on the keyboard is sticky.
taystrun@reddit
What is your preferred documentation tool, if you don't mind sharing?
-King-K-Rool-@reddit
Right now I just use OneNote, its what my org had before I showed up and I havent bothered to transfer everything over. Its not as bad as it used to be since they revamped it and I do like that it has a phone app and tells you who wrote each note.
Sleepytitan@reddit
I have 15 years experience in manufacturing IT, just took a similar job to OP, and could not have said it better. This is exactly what I’m doing and have done in previous roles.
I’ll add from experience, in manufacturing you’re going to find old systems. The sector is resistant to change and has to be forced to move forward sometimes.
You will need to build relationships and earn trust so that when it comes time to replace that XP machine that’s been running a critical process for 20 years, they’ll let you go through the painful process of making it happen.
m5daystrom@reddit
I started in manufacturing IT 40 years ago. Of course my dad's business had no IT so I had to come up with it. Intel 8088 until we got a 286 and was able to run SCO Xenix with dumb terminals!!!
Unique_Inevitable_27@reddit
Start with basics like inventory, backups, and access control, then improve things step by step. Also keep notes and use your mentor time wisely
Adept_Chemist5343@reddit
Documentation, backup and Standardization is your friend. I found the book The Practice of System and Network Administration an amazing resource
eggbeater98@reddit
Please feel free to PM me! I have been in this exact same scenario. Everyone else is saying the right things: figure out all your network hardware (and make sure your UPS batteries are thriving), VLANs, server structure, backups, passwords (use a password vault if there isn't already one. KeePass is great to start with and you can import a KeePass file into just about anything else you decide to use), and get familiar with the AD structure, GPOs, and M365 environment. Figure out service agreements and contracts. Are there printers?
Down the line: segregate networks, set up RBAC for everything, set up monitoring for network appliances, make sure you have reminders of when contracts are due to be renewed, check WiFi health, etc.
Security is paramount. Depending on where your factory is in the supply chain, there may be expectation for NIST compliance. Regardless, NIST is a good framework to have anyway.
Ohmystory@reddit
Document everything and start checking with how data are back up … as this your life line to recover data should things happen …
If there are gaps or compliance exposure let mgmt know and figure a method to improve …
Bluetooth_Sandwich@reddit
This, you should take the first 6-8 months gathering information before making any changes outside of what mgmt has delegated. I would also have a conversation with mgmt that you need to time to review what existing documentation is there (if any) and process your scope. I imagine you weren't given a scope sheet so that's where I would start first.
You got this, trust that they had no one in the gap between you and the last guy and so they're relieved to have someone in the role now.
darkraven1313@reddit
THIS! Number one goal should be backups. Make sure all critical data is being backed up regularly, and for the love of al that is holy do a test restore at least once a month to verify they are good.
Once that is squared away document everything. Personally I would start with hardware and critical business processes. Try to get management on board with a hardware refresh cycle. Nothing is worse then trying to keep a 12 year old server limping along because, "It works why do we need to replace it?". Asking what would happen (how much money would be lost) if the server was unavailable for 2-14 days while it is rebuilt on new hardware helps to highlight the risk.
GradeAccomplished322@reddit
Some objectives:
* Start a list of questions for the outsourced guy to ask during his limited availability windows in order of priority
* Start your own documentation - whether that's making a runbook or something is up to you. Assume you might have to share this documentation with smart hands (your outsourced IT contact or a useful and trusted company associate) in order to aid them if you're indisposed, so it needs to be readable within reason.
* Ensure your workstation is properly secured physically and important stuff is backed up in some manner (Google Drive, Microsoft OneDrive, whatever)
* Map the network, locate critical components (devices at the edge, core switches, servers, racks, etc). Ensure you have working credentials for everything. If you need physical keys or keycards or whatever, ensure you have those.
* Investigate you have access to all crucial services (Microsoft 365 admin, Google Workspace admin, domain registrar, ISP, website, whatever). Find out how they're billed and paid. Put it on your work calendar. It's probably mostly automated but you don't want to find out 30 days after a bill is missed.
* Start checking on backups. Find out what is backed up and how. Find out how restores work. Find out what isn't backed up and whether it should be.
* Start checking on updates. Find out what is updated and how. Find out what isn't updated and why.
* Start checking on IDS or other malware detection. Find out what product is in use and how it works.
* Find out what has MFA enabled and where that MFA goes. If something can have MFA enabled and it isn't enabled, see if you can determine why.
* Ask users about their jobs, what they use the computers for, how they use them, what the most critical functions and applications are. Are they working from home on weekends? Do they work on holidays or while travelling? Do they have company laptops? Are they allowed to use their own devices to access company resources?
* Review all documentation you've inherited
* Once you've learned a lot, you can start looking at improvements. Example: Are employees using personal devices to watch ESPN on the company wifi? Maybe you should have a guest wifi that's isolated they can use then.
TryTriGuy@reddit
Everything is pretty chill sounds good for now but if something essential goes wrong things will get un-chill very quickly!
I feel that you also need to manage expectations of your higher-ups. If you're busy on something but getting hassled about something else someone is going to have to wait. Don't let yourself get stressed about it, ask for help to prioritize then point anyone who is hassling you at your manager.
It's up to them to support you just make sure you keep them in the loop as to what's going on, if they're happy to help (as they should be) then all's good, if they're not then tough, they're in the loop so they'll know about anything that's going south and will have be behave like a manager whether they like it or not.
Don't be reluctant to use them as a resource, that's what they're there for whether they like it or not.
CaptainAdmiral85@reddit
Make a list of the entire tech stack. Build a home lab or office lab (just extra machines that are either already there and being unused or get them off eBay).
Use the list you make to watch How To Videos on how to setup and install / configure everything on that tech stack. Dedicate 3-4 hours a day to watching those videos and practicing on your home/office lab.
Learn automation. Learn how to automate things (scripting / dev ops) on all the operating systems you support. For Windows that would be Powershell for Linux focus on Python. Learn Docker, Kubernetes, Proxmox, Terraform, Puppet ....etc.
Make backups and Password Management your new religions. Every server you support (File Servers, Application Servers, Database Servers, Transaction Servers, Imaging Servers....etc) needs 3 backups. On prem, off site (different zip code) and the cloud (Backblaze or Crashplan or iDrive or Carbonite). You should be using a Password Manager like Bitwarden learn how to create an emergency kit of your own Bitwarden. Create an encrypted disk image and export all your password entries and 2FA tokens into it. Make 3 copies on USB Thumb Drives and keep them in 3 locations, one on your key chain, one in household safe and one in a bank safe deposit box. You need to update all 3 USB thumb drives every six months as over time you'll have new passwords and new 2FA tokens. This is to ensure you never get locked out of your digital accounts in case of fire that can destroy all your devices.
Also don't forget to have fun!
seanpmassey@reddit
I’m getting flashbacks to my first real full-time job when I was hired as the sole sysadmin for two cookie factories that were about 20 miles from my college. The two cookie factories were part of a much larger company, but I was the only IT guy onsite and the nearest IT support in my division were a 10 hour drive away. That was 20 years ago…and now I’m feeling old. LOL
So here’s my advice. First - DON’T CHANGE ANYTHING! And don’t recommend any changes to anything on the factory floor. At least not at first. You need to learn the environment. What they’re doing on the factory floor. What applications they are using. How they are doing their jobs. Changing anything without understanding this can lead to downtime, which is bad for a factory.
Don’t be surprised if some of the equipment you find and/or support is old. It won’t be replaced. Industrial equipment is expensive. One of the two factories I mentioned had 286s/386s running industrial machinery and ThinNET coax ethernet in 2005, and there were no plans to modernize it. And a maintenance system running on an old AS400 that was being shut down. And a label printing system that involved a combination of things you wouldn’t expect to hear in the same sentence: giant Zebra printers, a GUI built in Microsoft Access, an AS400, and JD Edwards.
Which brings me to my next point. Learn the business and understand how they make money. If you’re working in a low margin business, you won’t have a large IT budget and you’ll have to make do with whatever you have available.
As you settle into the role, spend time walking your network. Find out where the switches are. Find where the PCs on the factory floor are. Talk to the people using those PCs. Talk to the maintenance/facilities/production engineering people to find out what equipment they have and how it’s networked.
Finally, there will be this big temptation to improve things as you find issues. Don’t! Unplanned changes cause downtime. Downtime in a factory is ranges for annoying (a little over time to rework something) to “plant down, send everyone home.”.
No_Ring_3348@reddit
Best first move is to get disaster recovery setup properly, this will give you insight into processes, systems and people. Don't panic when you find old business-critical systems, e.g. I've had to support autoclaves that ran on Italian DOS 2.x and couldn't be virtualised, just document them extensively, ensure they're as robust as possible and have a 'in case of outage, break glass' system ready to replace it with minimal effort.
start_run_cmd@reddit
Fired from the NHS? - whatever happened must have been serious for that to actually happen! What did you do?! ;)
shimonj1@reddit (OP)
I forced myself to visit Thailand two times a year lol,
slippery_hemorrhoids@reddit
Like them ladyboys huh
Curious201@reddit
if you are solo IT for 300 people, i would treat the first few months as triage, not as “own everything immediately.” get backups verified first, not just “there is a backup job,” but an actual restore test for one file and one important system. then map the basics: network diagram, switches/firewalls, internet circuits, server list, admin accounts, licensing, vendor contacts, printers, critical apps, and who approves access. after that, build a simple ticket queue and stop letting everything arrive through hallway requests or random chats. you will still be busy, but at least you can show what is coming in, what is risky, and what needs money or outside help. one person can keep a factory limping along for a while, but you need documentation and priorities fast or every old hidden problem becomes “your” failure.
Avas_Accumulator@reddit
I was there. The answer was to use the solutions in Business Premium for everything, though you might need E3 now that/if you are 300+ devices/employees with a PC
It really is an all-in-one package to get started. Ensure you use Intune with Cloud-Only devices. I also recommend no legacy on-prem AD for a small company that "does not actually need it" if that is true. Then automate a lot of tasks in Logic Apps or Power Automate.
We tacked on CrowdStrike MDR and Check Point HEC for security to ensure operations is flowing. Then cross all points off in Microsoft Secure Score.
You now have a smooth and secure running baseline to work on
Thin_Weekend9564@reddit
Is this an Ice Cream factory🤡
DarkSky-8675@reddit
If it’s a factory you have an OT network. Document it.
Pure_Fox9415@reddit
Are there ERP, CRM, WMS, logustics, financial systems? Are they SaaS or on-premise? Where its databases and backups? Are there physical or virtual on-premise servers? Find it and setup backup of everything with any tool you have right now (robocopy/rsync for files, cloud connectors for saas content, built-in dump tools for dbs, etc).
Get access to routers find out network structure if any exists, scan with nmap all vlans/subnets, save this data. Backup network devices configs. DO NOT CHANGE ANYTHING AT THIS STAGE. even if it's a password for critical account "123456".
Start physical inventory of IT assets manually or with some ITAM tools like OCSinventory-ng.
Talk with management, what are their expectations for IT (my guess is they not even care to think about it too much, right now they need it to "just work" and do not know what they can get in terms of automation and business optimization).
After that, modernize and stabilize bare minimum of critical things, if needed, make it HA. Ensure you have full control and access to everything you need.
When everything shines polished, start to develop IT as a support for business - talk to employees, search what can be improved, improve with low-budget/free tools.
After some examples of success, talk to management to discuss a budget for improvements with commercial tools/services uf needed.
BurlyKnave@reddit
Do some research on finding a great book or three about the stuff you are managing
Reference books, how to books, for dummies books, what ever works for you
timmy_the_large@reddit
I love how I have always worked at a medium size company. First real company was about 200-250, the next was about 800, and now I am at a place a bit over 3000. I always felt I worked at a medium company.
JoeMiner79@reddit
Best opportunity in your life to learn, congrats! 👍🏻
TriggernometryPhD@reddit
I give it 30 days.
ArborlyWhale@reddit
Follow your countries guidelines for cybersecurity. Canada has https://www.cyber.gc.ca/en/cyber-security-readiness/cross-sector-cyber-security-readiness-goals-toolkit
The-IT_MD@reddit
This company didn’t care about its IT. Sorry OP.
Apprehensive-Care341@reddit
Get a ticketing system and email address
recovering-pentester@reddit
Think you need a mentor that’s available for more than 1 day a week lol.
Tbf, I’d be in the same boat!
Ethernetman1980@reddit
I was in the same boat about 10 years ago and it turned out to be a great job. Backups first make sure they are working onsite and offsite. Understand how a proper 3-2-1 backup is setup and get help if you need it. 2nd start thinking about implementing redundancy when you have downtime. Most factories I've worked for run 24/7 but if you can have a new server or VM ready to go that will help. Check low hanging fruit like EDR software, MFA on email, local Admin rights, Password Requirements. If Budget permits start replacing machine with the best quality machine you can get approval for. Users are less likely to bother you when there Excel, or whatever runs fast. I remember the first thing I did was got rid of an on prem Exchange Server the 2nd thing I did was upgrade the DC/File/Print Server from 2003. Don't forget Internet speed either some places forget to upgrade and are paying for something from 5-10 years ago when they could pay the same price today for 3 or 4x the internet speed. Offload printer / copier supplies and repair if you can - That's a pain in the ass you don't want to deal with. In fact, I would start replacing anything older than 5 years or 150k copies. Get an AI subscription I use Claude and it streamlines a lot of research I would have spent days on 10 years ago.
lornranger@reddit
What’s the scope of your job?
KillingTime1212@reddit
First get a grasp on where the important data lives and make sure it’s backed up daily. Make sure you can restore it to a test location. Then make sure it’s being backed up to a cloud location that is immutable. This includes network, firewall, and phone system config backups.
Then spend time documenting everything.
You don’t need to change anything unless it’s critical. Just get a feel for the land.
libertyprivate@reddit
I suppose first you come up with a question, then you ask it