| livejournal sux, here's a shitty workaround |
[Dec. 4th, 2008|03:43 pm] |
I can't post a reply over 4300 characters in someone else's journal, so here's a reply here without dumbass arbitrary limits. Not of interest to others, but since I've got some other security/hacker practitioners as friends who read this, you're welcome to offer your comments too. This is just my perspective having been in a role of errr support of some sort for . . . too fucking long. Albeit, my views outline an ideal, and I've certainly dealt with things that put a lot more burden on me than I like handling and I deal with that; it's also a goal to not put too much burden on users [that doesn't tend to work or scale], but where do developers fit in? If you are interested in my biased thoughts, read on:
I don't think it necessarily fits into your two choices of voting; but even though I tend to do more network/sysadmin/security stuff I guess that's me. Albeit, I do and have done lots of network/system security work in the past. My philosophy is something along the lines of:
do you have root/admin on the machine?
If yes: Then you better be responsible and keep things patched, you want the 'power' then you don't get to get away scott free. Albeit, you are probably not the only one with this responsibility, if there's an admin team at your job that has root, you all get to share in it. When giving friend's a shell on a machine I own, I tend to automatically give them root [because most of my friends could get it easily if I didn't provide it anyway] and tell them they get to pitch in with sysadmin duties. This tends to work well as long as everyone in wheel is communicating with each other. Some of this is reflected by a friend of mine (Bob Beck of UoA) who has often turned young snotty hacker students who own one of the machines on campus into his unofficial minions. How did he put it? Something along the lines of, "I feign being impressed by their skills long enough to please their sense of ego, then I ask them if they can help me out and teach me some of their amazing skills by showing me how they keep a system secure. And eventually, I have them wrapped around my pinky finger doing whatever I ask them to do." Beck is an evil evil evil man, but also a great hacker and realist; but most kids don't understand the idea of being manipulated on that level, at all. ;) His page is here: http://www.ualberta.ca/~beck/ and the character from Brazil is really pretty much the personification of bob and other top notch ops guys I've been lucky enough to run into or work with in my career. Anyway enough on that...
If you don't have root: not your job, if sysadmin slacks, that's their fuckup. Believe me, as a security guy sometimes I don't -want- access to your system, I don't want the finger pointed at me for your fuckups. I mean sure, I can -get- access to your system, but if it's not provided for me (or if I don't tell people I'm on it) they can't blame me when it's broken. Same goes for you to systems you don't have control over!
But as far as the role of the developer. There are often hard battle lines between ops and development; where ops people don't -want- developers to have any kind of root/admin privileges. Or if they give it, they do so very sparingly or with a watchful eye. There's generally some history behind this [from both sides, but let me talk about an ops perspective]:
The worst security compromise I've dealt with were on machines that developers ran in a sandbox network I setup for them to do whatever they wanted with admin rights (windows shop mostly there). They didn't patch their shit, and they also bridged into the production network because they were tired of sneakernetting or vpning things to the sandbox development network and ended up letting loose a worm on the internal production LAN. To make matters more worse, there were other machines on the network that got infected which were machines that -they- had taken it upon themselves to put in place so users could use beta versions of their software, and again they didn't follow standard build+patching procedures and were outside the scope of what our very limited IT staff had to deal with (we were about 10X the size of kink, and had about half the amount of ops staff). They said they *needed* admin access and we gave it to them with the knowledge that they were responsible for keeping their machines patched as our bargain. That was a fair deal, they got their autonomy and were happy they could do whatever they wanted, and we were happy because we had less to deal with and didn't need to bugfix crazy alpha builds of whatever crappy stuff they wanted to run. The reality was we gave them those rights, and they didn't keep up their end of the bargain. Cleaning up other people's shit is a reality in any sort of ops job, but when you're cleaning up other people's shit who said they could wipe their own ass and you put your trust in them and the shit all over your world, it's infuriating. It's much worse than the clueless user who doesn't know better; I wanted to see people fired for negligence. (This did happen eventually not as a result of that, but I'm sure it didn't help their case much that they were clearly jerkoffs who liked a lot of toys but didn't like to pick up after them, I'm obviously still bitter).
That said, I've worked in worse scenarios too - when I worked at the county office of education in SLO, it was me, another guy and our boss as IT for 45+ sites throughout the county. Any problem that came up was ours to deal with from helping people check their email to becoming the first county to join up to the Digital California Project [a subsidized high speed network for california school districts by county]; and that kind of 3:several thousands to support just doesn't scale. If you found a user who had a clue, you would do everything you could to educate them to take care of themselves and ideally even people they worked near (traveling between sites occupied much of many days). We certainly never faulted anyone for not taking care of things if they couldn't, but if they could - it was a huge help! Most educational environments are like this - there are too many things going on for the ops guys to fight every fire, and if you are a developer and writing software an overtaxed ops guy is going to assume/or at least HOPE you have a basic clue and can do fundamental routine maintenance on your system assuming you have the privileges. In that environment I often would freely give out admin/root privileges to people who had a clue so that they -could- install their own patches and updates, by simply asking me for that ability they'd demonstrated enough for me to give them the benefit of the doubt that even if they did shoot themselves in the foot later they were taking steps to make sure that wouldn't happen. There are plenty of people who know a lot less about technology than an average "developer" and naturally are going to necessitate more attention, but hey you write software, you should know a thing or two.
There used to be (or IMNSHO *should* be) a progression:
neophyte -> user -> power user -> admin -> developer (EDIT:) -> wizard/hacker.
The reality is on the higher end of things (be it admin work or development work) there's a lot of crossover. A real wizard handles everything themselves and generally only asks other wizards for pointers on how to do things better.
Sadly developers (and even many CS professors) these days never even get to the power user stage; computers are great symbolic abstraction tools, but if you have your head in the clouds of whatever newfangled fad language/api/library and don't understand the fundamentals of discrete systems and the layers you're working on top of (one of those previous horrid developers I mentioned above was writing webapps, and didn't even understand how to use ping to check for connectivity to a system he was trying to poll from) I'm of the mindset that you will pretty much never make things that are satisfying for people to work with. On a bad day, this is when I talk about building a robot army to kill all humans in reference to such people.
Anyway a lot of this is opinion, but the main thing in dealing with this situation that I would recommend comes from the message you quoted:
"If you need help, I can put you in touch with the campus sns team."
I would do that. They're probably more than happy to do hand holding, or the entire patching process for you. A good admin/security person will know how to back things up to rollback onto if needed, or realize when you need to push forward and fix some bugs/config issues to deal with the 'new world' despite the breakage. If you have a team there as a resource, it seems far more effective to use them as that's been provided an option than to worry about what is being asked of you, since you've got an option to foist things onto someone else.
That said too... most patches are designed to be minimally intrusive and compatible, things that will cause problems are generally explicitly stated in the patch release and you should review them before running with them and see if anything jumps out at you. If it's all nonsense to you, chances are it's not something you need to worry about anyway. And, if you're sufficiently behind in your patching, if you do patch and it gets hairy, you can generally rely on the internet to find someone who ran into the same issue as well as a fix or workaround.
Sadly the security of a system doesn't just rely on it being 'patched.' For example, the number of sql injection bugs introduced by developers who don't understand step 1 about security or sane ways to interact with a database from an untrusted input is very trying. I know some great brilliant people out there who stop developing software because they simply stopping wanting to be responsible for trying to figure out every edge case that could lead to catastrophe and found far more satisfying technical lives in hardware design [albeit errors in hardware design can be far far more costly, but a lot more time is spent in sanity checking things too].
Even a good hacker/security practitioner is constantly fighting a battle of proportions where any 13 year old with a copy of the 1337h4X0r tool du jour can do serious damage with minimal skills or understanding. The odds are already against the security people and they can't do it alone - in any place I've worked or college curriculum I've helped design, I've encouraged educating not just the security practitioners but the public at large. Patching should be like changing the oil or tires on your car, help keep it running optimally and reduce the likelihood of a catastrophic breakdown, it's just routine maintenance and shouldn't break things (or in cases where it does it's often because the software was written in a manner that was insecure to begin with and that practice should change). And most of all, don't be scared of it - know how to do these things yourself, even if you discover you don't like changing the tire on your car yourself, and get it serviced by people who do that for a job usually, having done it at least can help you in a bind and save you a lot of headaches. (to use an analogy)
Also for 'staging' that's what vm's are great for these days. IMNSHO all development should start in a vm environment before ever being pushed live (and in some cases vm's make sense for live deployments too). But you may not have those resources at your disposal. You mentioned apache and other very standard F/OSS tools - and there are other F/OSS tools to help you do just that, learning about them will only add useful things to your resume for the future and make you a more well rounded and capable.
That said, I understand this stuff can be a drag; turns out it is for everyone. But where the finger is pointed or responsibility foisted doesn't change that for whoever does the gruntwork no matter "who's job" it is. And again, like I said - if you've got a team at your disposal to lend a hand, use them. The drag at least can be distributed among more people, and you can probably learn some things or best practices from them in the long run. There can be tradeoffs with security practices (often in convenience) but at least being aware of the process can help you, and help you realize when maybe there's actually a more convenient way of doing something that is more secure too (e.g. using ssh public keys instead of telnet or something). |
|
|
![[info]](http://l-stat.livejournal.com/img/userinfo.gif){kind=small}
artkiver's journal
